[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:56:58 dist-test-slave-nj21 krb5kdc[21626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:56:58 dist-test-slave-nj21 krb5kdc[21626](info): set up 2 sockets
Aug 08 06:56:58 dist-test-slave-nj21 krb5kdc[21626](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 06:56:59.914861 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:56:59 dist-test-slave-nj21 krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:56:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:00.027954 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:41961
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:41961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:00.110751 21647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:00.110957 21647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:00.110978 21647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:00.112110 21647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:00.112149 21647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:00.112161 21647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:00.112172 21647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:00.112182 21647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:00.113786 21647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:41961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:41961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:00.114128 21647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:00.114348 21647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:00.121806 21665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:00.121889 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:00.122216 21661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:00.122351 21647 server_base.cc:1034] running on GCE node
I20260808 06:57:00.122645 21647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:00.123035 21647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:00.124171 21647 hybrid_clock.cc:648] HybridClock initialized: now 1786172220124167 us; error 53 us; skew 500 ppm
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:00.126662 21647 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:00.127279 21647 webserver.cc:460] Webserver started at http://127.21.28.126:33147/ using document root <none> and password file <none>
I20260808 06:57:00.127549 21647 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:00.127606 21647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:00.127792 21647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:00.128947 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "73b816341314462e8ef2a3641d31129e"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "9f7b723466814936018f870c324f0780"
server_key_iv: "ad31a73d38ae91b8ed59d609200c483b"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.129307 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "73b816341314462e8ef2a3641d31129e"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "9f7b723466814936018f870c324f0780"
server_key_iv: "ad31a73d38ae91b8ed59d609200c483b"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.131158 21647 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:57:00.132164 21677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.132402 21647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:00.132494 21647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "73b816341314462e8ef2a3641d31129e"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "9f7b723466814936018f870c324f0780"
server_key_iv: "ad31a73d38ae91b8ed59d609200c483b"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.132552 21647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:00.149432 21647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:00.151633 21647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:00.151829 21647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:00.156906 21647 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:41961
I20260808 06:57:00.156976 21771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:41961 every 8 connection(s)
I20260808 06:57:00.157544 21647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:00.158921 21772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:00.162114 21772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e: Bootstrap starting.
I20260808 06:57:00.162637 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 21647
I20260808 06:57:00.162719 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:00.163156 21772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:00.163348 21617 external_mini_cluster.cc:1428] Setting key b551581e4cab631c2ba5ad2618652daa
I20260808 06:57:00.163534 21772 log.cc:824] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:00.164444 21772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e: No bootstrap required, opened a new log
I20260808 06:57:00.166397 21772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b816341314462e8ef2a3641d31129e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41961 } }
I20260808 06:57:00.166569 21772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:00.166664 21772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b816341314462e8ef2a3641d31129e, State: Initialized, Role: FOLLOWER
I20260808 06:57:00.166852 21772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b816341314462e8ef2a3641d31129e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41961 } }
I20260808 06:57:00.167008 21772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:00.167050 21772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:00.167104 21772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:00.168120 21772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b816341314462e8ef2a3641d31129e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41961 } }
I20260808 06:57:00.168351 21772 leader_election.cc:302] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73b816341314462e8ef2a3641d31129e; no voters:
I20260808 06:57:00.168562 21772 leader_election.cc:288] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:00.168601 21777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:00.168813 21777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [term 1 LEADER]: Becoming Leader. State: Replica: 73b816341314462e8ef2a3641d31129e, State: Running, Role: LEADER
I20260808 06:57:00.169004 21777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b816341314462e8ef2a3641d31129e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41961 } }
I20260808 06:57:00.169131 21772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:00.169607 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73b816341314462e8ef2a3641d31129e. Latest consensus state: current_term: 1 leader_uuid: "73b816341314462e8ef2a3641d31129e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b816341314462e8ef2a3641d31129e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41961 } } }
I20260808 06:57:00.169669 21778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73b816341314462e8ef2a3641d31129e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b816341314462e8ef2a3641d31129e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41961 } } }
I20260808 06:57:00.169776 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:00.169785 21778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:00.170121 21790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:00.171329 21790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:00.173668 21790 catalog_manager.cc:1347] Generated new cluster ID: b7aee232a094451d9d19ae3ceb8c4097
I20260808 06:57:00.173723 21790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:00.179348 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.164708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.21.28.126:41961)
0808 06:57:00.165033 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:00.165044 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:00.165209 (+ 165us) server_negotiation.cc:407] Connection header received
0808 06:57:00.166988 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:00.166995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:00.167072 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:00.167207 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:00.168919 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.170152 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.171765 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.172357 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.172470 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.176053 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:00.176057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:00.176057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:00.176068 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:00.177596 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.177984 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.177986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.177986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.178076 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.178306 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.178307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.178308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.178528 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:00.178682 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:00.178939 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 06:57:00.179116 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260808 06:57:00.187563 21790 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:00.188117 21790 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:00.196749 21790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73b816341314462e8ef2a3641d31129e: Generated new TSK 0
I20260808 06:57:00.196913 21790 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:00.226496 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41961
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:00.343348 21812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:00.343544 21812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:00.343567 21812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:00.344745 21812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:00.344790 21812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:00.344815 21812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:00.346266 21812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41961
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:00.346562 21812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:00.346792 21812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:00.352686 21824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:00.352713 21823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:00.352942 21827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:00.353039 21812 server_base.cc:1034] running on GCE node
I20260808 06:57:00.353224 21812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:00.353434 21812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:00.354575 21812 hybrid_clock.cc:648] HybridClock initialized: now 1786172220354560 us; error 32 us; skew 500 ppm
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:00.357605 21812 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:00.358135 21812 webserver.cc:460] Webserver started at http://127.21.28.65:35305/ using document root <none> and password file <none>
I20260808 06:57:00.358430 21812 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:00.358502 21812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:00.358644 21812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:00.360069 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "c5271bc7402c4b028a50b5e8ea2975cc"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "dba8eb8782e89d43a51f85fcef5d7bad"
server_key_iv: "9209354b9bdef047e696bb5a7b692589"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.360522 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "c5271bc7402c4b028a50b5e8ea2975cc"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "dba8eb8782e89d43a51f85fcef5d7bad"
server_key_iv: "9209354b9bdef047e696bb5a7b692589"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.362392 21812 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260808 06:57:00.363386 21839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.363582 21812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:00.363713 21812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "c5271bc7402c4b028a50b5e8ea2975cc"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "dba8eb8782e89d43a51f85fcef5d7bad"
server_key_iv: "9209354b9bdef047e696bb5a7b692589"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.363795 21812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:00.384838 21812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:00.386587 21812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:00.386735 21812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:00.387053 21812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:00.387460 21812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:00.387542 21812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.387578 21812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:00.387600 21812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.396124 21812 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:40243
I20260808 06:57:00.396165 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:40243 every 8 connection(s)
I20260808 06:57:00.396463 21812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:00.401912 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 21812
I20260808 06:57:00.401988 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:00.402112 21617 external_mini_cluster.cc:1428] Setting key f182c1ada8c2b7698f35afd6c5775187
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:00.406057 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.397381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:54867 (local address 127.21.28.126:41961)
0808 06:57:00.397433 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:00.397435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:00.397627 (+ 192us) server_negotiation.cc:407] Connection header received
0808 06:57:00.398129 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:00.398130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:00.398144 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:00.398195 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:00.399261 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.399818 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.400772 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.400983 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.401039 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.403253 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:00.403255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:00.403255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:00.403263 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:00.404675 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.405143 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.405144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.405145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.405198 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.405443 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.405443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.405444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.405537 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:00.405580 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:00.405886 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 06:57:00.405959 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 06:57:00.406219 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.397340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:41961 (local address 127.21.28.65:54867)
0808 06:57:00.397514 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:00.397548 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:00.398027 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:00.398288 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:00.398292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:00.398974 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:00.399160 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:00.399165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.399907 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.399908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:00.400679 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:00.400681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.401053 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.401054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:00.401163 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.401609 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:00.401617 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:00.403166 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:00.404767 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:00.404775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:00.404778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:00.405062 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:00.405250 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:00.405251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:00.405251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:00.405387 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:00.405585 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:00.405590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:00.405756 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 06:57:00.405872 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 06:57:00.406016 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260808 06:57:00.406818 22015 heartbeater.cc:347] Connected to a master server at 127.21.28.126:41961
I20260808 06:57:00.406905 22015 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:00.407068 22015 heartbeater.cc:511] Master 127.21.28.126:41961 requested a full tablet report, sending...
I20260808 06:57:00.408026 21707 ts_manager.cc:194] Registered new tserver with Master: c5271bc7402c4b028a50b5e8ea2975cc (127.21.28.65:40243)
I20260808 06:57:00.409082 21707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:54867
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:00.430251 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41961
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:00.535144 22024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:00.535336 22024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:00.535358 22024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:00.536491 22024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:00.536535 22024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:00.536559 22024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:00.538075 22024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41961
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:00.538266 22024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:00.538439 22024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:00.543548 22042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:00.543550 22032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:00.543660 22036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:00.543993 22024 server_base.cc:1034] running on GCE node
I20260808 06:57:00.544158 22024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:00.544373 22024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:00.545506 22024 hybrid_clock.cc:648] HybridClock initialized: now 1786172220545482 us; error 37 us; skew 500 ppm
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:00.548439 22024 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:00.548909 22024 webserver.cc:460] Webserver started at http://127.21.28.66:38757/ using document root <none> and password file <none>
I20260808 06:57:00.549101 22024 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:00.549140 22024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:00.549242 22024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:00.550312 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "2996fc58a6e6424bb3b26affd4ce61b9"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "b16c14d042dacae7a187f2514ae033ff"
server_key_iv: "3b65a211e589d1efe17b6bbdcb10470c"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.550649 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "2996fc58a6e6424bb3b26affd4ce61b9"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "b16c14d042dacae7a187f2514ae033ff"
server_key_iv: "3b65a211e589d1efe17b6bbdcb10470c"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.552039 22024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 06:57:00.552755 22050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.552909 22024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.552990 22024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "2996fc58a6e6424bb3b26affd4ce61b9"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "b16c14d042dacae7a187f2514ae033ff"
server_key_iv: "3b65a211e589d1efe17b6bbdcb10470c"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.553043 22024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:00.563633 22024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:00.565668 22024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:00.565807 22024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:00.566063 22024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:00.566393 22024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:00.566429 22024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.566449 22024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:00.566471 22024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.578012 22024 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:41045
I20260808 06:57:00.578075 22217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:41045 every 8 connection(s)
I20260808 06:57:00.578379 22024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:00.585210 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22024
I20260808 06:57:00.585291 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:00.585436 21617 external_mini_cluster.cc:1428] Setting key 9b463efa68f0e0cd8badd87b60ca19d5
I20260808 06:57:00.587845 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.579145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:39513 (local address 127.21.28.126:41961)
0808 06:57:00.579202 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:00.579203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:00.579479 (+ 276us) server_negotiation.cc:407] Connection header received
0808 06:57:00.580282 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:00.580283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:00.580297 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:00.580346 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:00.581400 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.581895 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.582944 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.583136 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.583192 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.585326 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:00.585328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:00.585328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:00.585335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:00.586617 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.586971 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.586972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.586972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.587029 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.587242 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.587243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.587243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.587351 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:00.587407 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:00.587652 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 06:57:00.587755 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 06:57:00.588006 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.579118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:41961 (local address 127.21.28.66:39513)
0808 06:57:00.579352 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:00.579386 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:00.580185 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:00.580412 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:00.580415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:00.581063 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:00.581303 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:00.581308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.581975 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.581976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:00.582861 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:00.582863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.583203 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.583204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:00.583371 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.583924 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:00.583931 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:00.585233 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:00.586691 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:00.586697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:00.586699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:00.586906 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:00.587083 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:00.587083 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:00.587083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:00.587188 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:00.587406 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:00.587412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:00.587539 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 06:57:00.587624 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 06:57:00.587794 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":121,"threads_started":1}
I20260808 06:57:00.588402 22218 heartbeater.cc:347] Connected to a master server at 127.21.28.126:41961
I20260808 06:57:00.588492 22218 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:00.588650 22218 heartbeater.cc:511] Master 127.21.28.126:41961 requested a full tablet report, sending...
I20260808 06:57:00.588997 21707 ts_manager.cc:194] Registered new tserver with Master: 2996fc58a6e6424bb3b26affd4ce61b9 (127.21.28.66:41045)
I20260808 06:57:00.589461 21707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:39513
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:00.611286 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41961
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:00.710971 22228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:00.711160 22228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:00.711182 22228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:00.712323 22228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:00.712364 22228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:00.712389 22228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:00.713979 22228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41961
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:00.714157 22228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:00.714340 22228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:00.720238 22238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:00.720263 22240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:00.720288 22237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:00.720439 22228 server_base.cc:1034] running on GCE node
I20260808 06:57:00.720690 22228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:00.720929 22228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:00.722079 22228 hybrid_clock.cc:648] HybridClock initialized: now 1786172220722064 us; error 32 us; skew 500 ppm
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:00.724767 22228 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:00.725142 22228 webserver.cc:460] Webserver started at http://127.21.28.67:33867/ using document root <none> and password file <none>
I20260808 06:57:00.725307 22228 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:00.725342 22228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:00.725440 22228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:00.726413 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "798b05ce73044135b7da783813c8d56a"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "bf4f4f313757b34ca3bf9442f7c1283b"
server_key_iv: "0c74f8066683b0985e5f5e990e15a0e4"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.726701 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "798b05ce73044135b7da783813c8d56a"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "bf4f4f313757b34ca3bf9442f7c1283b"
server_key_iv: "0c74f8066683b0985e5f5e990e15a0e4"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.728076 22228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:00.728785 22254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.728943 22228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.729033 22228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "798b05ce73044135b7da783813c8d56a"
format_stamp: "Formatted at 2026-08-08 06:57:00 on dist-test-slave-nj21"
server_key: "bf4f4f313757b34ca3bf9442f7c1283b"
server_key_iv: "0c74f8066683b0985e5f5e990e15a0e4"
server_key_version: "encryptionkey@0"
I20260808 06:57:00.729089 22228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:00.741547 22228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:00.743352 22228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:00.743487 22228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:00.743819 22228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:00.744131 22228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:00.744164 22228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.744184 22228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:00.744200 22228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:00.757782 22228 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:39131
I20260808 06:57:00.757841 22415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:39131 every 8 connection(s)
I20260808 06:57:00.758138 22228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:00.766108 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22228
I20260808 06:57:00.766172 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:00.766296 21617 external_mini_cluster.cc:1428] Setting key 9565651b1d7d99668995be68ddeb0211
I20260808 06:57:00.767647 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.758983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:34679 (local address 127.21.28.126:41961)
0808 06:57:00.759040 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:00.759042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:00.759294 (+ 252us) server_negotiation.cc:407] Connection header received
0808 06:57:00.760006 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:00.760007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:00.760016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:00.760068 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:00.761111 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.761594 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.762565 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.762735 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.762776 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.765029 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:00.765030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:00.765031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:00.765037 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:00.766328 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.766695 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.766697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.766697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.766760 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.767042 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.767042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.767043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.767182 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:00.767248 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:00.767516 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 06:57:00.767582 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 06:57:00.767931 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.758953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:41961 (local address 127.21.28.67:34679)
0808 06:57:00.759159 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:00.759201 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:00.759926 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:00.760126 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:00.760129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:00.760765 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:00.761044 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:00.761049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.761654 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.761655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:00.762505 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:00.762507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.762785 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.762785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:00.762895 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.763360 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:00.763367 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:00.764938 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:00.766413 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:00.766419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:00.766420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:00.766619 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:00.766821 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:00.766822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:00.766822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:00.766972 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:00.767248 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:00.767254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:00.767394 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 06:57:00.767497 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 06:57:00.767648 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260808 06:57:00.768416 22416 heartbeater.cc:347] Connected to a master server at 127.21.28.126:41961
I20260808 06:57:00.768507 22416 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:00.768654 22416 heartbeater.cc:511] Master 127.21.28.126:41961 requested a full tablet report, sending...
I20260808 06:57:00.768959 21707 ts_manager.cc:194] Registered new tserver with Master: 798b05ce73044135b7da783813c8d56a (127.21.28.67:39131)
I20260808 06:57:00.769361 21707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:34679
I20260808 06:57:00.778127 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:00.803293 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.795550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.21.28.65:40243)
0808 06:57:00.795774 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:00.795779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:00.795792 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:57:00.795818 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:00.795820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:00.795835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:00.795904 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:00.796500 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.797411 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.798129 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.798501 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.798564 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.800705 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:00.800713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:00.800713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:00.800720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:00.802175 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.802471 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.802474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.802474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.802525 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.802689 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.802689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.802690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.802813 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:00.803050 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:00.803098 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 06:57:00.803172 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20260808 06:57:00.803795 21878 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56528
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[21626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:00.826922 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:00.820921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.21.28.65:40243)
0808 06:57:00.820983 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:00.820985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:00.821044 (+ 59us) server_negotiation.cc:407] Connection header received
0808 06:57:00.821082 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:00.821083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:00.821092 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:00.821135 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:00.821594 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.822097 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.822744 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:00.822927 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:00.822967 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:00.824889 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:00.824890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:00.824890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:00.824896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:00.826052 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.826338 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.826339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.826339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.826396 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:00.826532 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:00.826532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:00.826532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:00.826623 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:00.826710 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:00.826770 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 06:57:00.826835 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 06:57:00.827428 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 21812
I20260808 06:57:00.833988 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22024
I20260808 06:57:00.841087 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22228
I20260808 06:57:00.849255 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 21647
2026-08-08T06:57:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2077 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[22435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[22435](info): set up 2 sockets
Aug 08 06:57:00 dist-test-slave-nj21 krb5kdc[22435](info): commencing operation
krb5kdc: starting...
W20260808 06:57:01.972208 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:02.082101 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46569
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:02.160507 22452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:02.160701 22452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:02.160728 22452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:02.161846 22452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:02.161892 22452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:02.161912 22452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:02.161928 22452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:02.161943 22452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:02.163378 22452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:02.163597 22452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:02.163815 22452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:02.168422 22461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:02.168414 22464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:02.168416 22460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:02.168967 22452 server_base.cc:1034] running on GCE node
I20260808 06:57:02.169153 22452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:02.169384 22452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:02.170516 22452 hybrid_clock.cc:648] HybridClock initialized: now 1786172222170500 us; error 30 us; skew 500 ppm
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:02.172838 22452 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:02.173241 22452 webserver.cc:460] Webserver started at http://127.21.28.126:33301/ using document root <none> and password file <none>
I20260808 06:57:02.173450 22452 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:02.173492 22452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:02.173606 22452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:02.174558 22452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "a294c48a67ea458bafc01354ee284915"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "fe0b04d65ebeca177c3e5021ea1ba2dc"
server_key_iv: "304a41433f062fddac31cbbc4d8d1e7c"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.174876 22452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "a294c48a67ea458bafc01354ee284915"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "fe0b04d65ebeca177c3e5021ea1ba2dc"
server_key_iv: "304a41433f062fddac31cbbc4d8d1e7c"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.176132 22452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:02.176733 22473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.176885 22452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:02.176967 22452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "a294c48a67ea458bafc01354ee284915"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "fe0b04d65ebeca177c3e5021ea1ba2dc"
server_key_iv: "304a41433f062fddac31cbbc4d8d1e7c"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.177022 22452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:02.188891 22452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:02.190438 22452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:02.190568 22452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:02.194279 22452 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46569
I20260808 06:57:02.194309 22548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46569 every 8 connection(s)
I20260808 06:57:02.194604 22452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:02.195375 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:02.195896 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22452
I20260808 06:57:02.195976 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:02.196099 21617 external_mini_cluster.cc:1428] Setting key d4212efc7494e03d56147a0bc03188f6
I20260808 06:57:02.198184 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915: Bootstrap starting.
I20260808 06:57:02.198814 22549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:02.199120 22549 log.cc:824] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:02.199795 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915: No bootstrap required, opened a new log
I20260808 06:57:02.201732 22549 raft_consensus.cc:348] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a294c48a67ea458bafc01354ee284915" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46569 } }
I20260808 06:57:02.201869 22549 raft_consensus.cc:374] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:02.201901 22549 raft_consensus.cc:729] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a294c48a67ea458bafc01354ee284915, State: Initialized, Role: FOLLOWER
I20260808 06:57:02.202019 22549 consensus_queue.cc:260] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a294c48a67ea458bafc01354ee284915" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46569 } }
I20260808 06:57:02.202093 22549 raft_consensus.cc:388] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:02.202126 22549 raft_consensus.cc:482] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:02.202169 22549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:02.202773 22549 raft_consensus.cc:504] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a294c48a67ea458bafc01354ee284915" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46569 } }
I20260808 06:57:02.202870 22549 leader_election.cc:302] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a294c48a67ea458bafc01354ee284915; no voters:
I20260808 06:57:02.203022 22549 leader_election.cc:288] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:02.203063 22556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:02.203254 22549 sys_catalog.cc:564] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:02.203431 22556 raft_consensus.cc:686] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [term 1 LEADER]: Becoming Leader. State: Replica: a294c48a67ea458bafc01354ee284915, State: Running, Role: LEADER
I20260808 06:57:02.203534 22556 consensus_queue.cc:237] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a294c48a67ea458bafc01354ee284915" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46569 } }
I20260808 06:57:02.203918 22557 sys_catalog.cc:455] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a294c48a67ea458bafc01354ee284915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a294c48a67ea458bafc01354ee284915" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46569 } } }
I20260808 06:57:02.203999 22557 sys_catalog.cc:458] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:02.204145 22561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a294c48a67ea458bafc01354ee284915. Latest consensus state: current_term: 1 leader_uuid: "a294c48a67ea458bafc01354ee284915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a294c48a67ea458bafc01354ee284915" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46569 } } }
I20260808 06:57:02.204198 22561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915 [sys.catalog]: This master's current role is: LEADER
W20260808 06:57:02.204916 22582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:57:02.204965 22582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:57:02.205385 22573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:02.205566 22573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:02.206570 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.196573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.21.28.126:46569)
0808 06:57:02.196749 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.196758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.196796 (+ 38us) server_negotiation.cc:407] Connection header received
0808 06:57:02.197265 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.197274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.197342 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.197491 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:02.198005 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.199306 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.200185 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.200589 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.200685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.202923 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:02.202927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:02.202927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:02.202939 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:02.204973 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.205313 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.205316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.205316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.205429 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.205667 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.205668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.205668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.205976 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:02.206134 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.206251 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.206384 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260808 06:57:02.207124 22573 catalog_manager.cc:1347] Generated new cluster ID: befa8644939b48ce8d746888868fb131
I20260808 06:57:02.207175 22573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:02.217024 22573 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:02.217706 22573 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:02.222597 22573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a294c48a67ea458bafc01354ee284915: Generated new TSK 0
I20260808 06:57:02.222749 22573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:02.248615 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46569
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:02.328385 22588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:02.328577 22588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:02.328608 22588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:02.329752 22588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:02.329792 22588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:02.329828 22588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:02.331288 22588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46569
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:02.331523 22588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:02.331748 22588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:02.336602 22604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:02.336609 22600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:02.336683 22599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:02.336655 22588 server_base.cc:1034] running on GCE node
I20260808 06:57:02.336963 22588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:02.337172 22588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:02.338320 22588 hybrid_clock.cc:648] HybridClock initialized: now 1786172222338311 us; error 26 us; skew 500 ppm
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:02.340685 22588 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:02.341053 22588 webserver.cc:460] Webserver started at http://127.21.28.65:43177/ using document root <none> and password file <none>
I20260808 06:57:02.341254 22588 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:02.341298 22588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:02.341424 22588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:02.342444 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "ec67db3437a94eb291f5bcafb7ac4e36"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "c12168a4c60ec3b6d9bab0c1aaadf0d6"
server_key_iv: "7bf261fc5359855d16b807bc1d408782"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.342764 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "ec67db3437a94eb291f5bcafb7ac4e36"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "c12168a4c60ec3b6d9bab0c1aaadf0d6"
server_key_iv: "7bf261fc5359855d16b807bc1d408782"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.344058 22588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:02.344719 22611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.344877 22588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:02.344954 22588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "ec67db3437a94eb291f5bcafb7ac4e36"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "c12168a4c60ec3b6d9bab0c1aaadf0d6"
server_key_iv: "7bf261fc5359855d16b807bc1d408782"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.345019 22588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:02.355387 22588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:02.357060 22588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:02.357183 22588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:02.357403 22588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:02.357702 22588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:02.357736 22588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.357756 22588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:02.357781 22588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.363442 22588 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:44397
I20260808 06:57:02.363500 22777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:44397 every 8 connection(s)
I20260808 06:57:02.363792 22588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:02.372226 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.364458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:54475 (local address 127.21.28.126:46569)
0808 06:57:02.364517 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.364519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.364703 (+ 184us) server_negotiation.cc:407] Connection header received
0808 06:57:02.365709 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.365710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.365719 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.365770 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:02.366313 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.366837 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.367816 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.368031 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.368078 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.370066 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:02.370067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:02.370068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:02.370075 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:02.371125 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.371481 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.371481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.371482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.371528 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.371728 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.371729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.371729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.371831 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:02.371890 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.372096 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.372162 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 06:57:02.372375 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.364437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46569 (local address 127.21.28.65:54475)
0808 06:57:02.364596 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:02.364630 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:02.365624 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:02.365822 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:02.365825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:02.366054 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:02.366229 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.366233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.366922 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.366923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.367747 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.367750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.368113 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.368114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.368220 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.368662 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:02.368670 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:02.369984 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:02.371196 (+ 1212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:02.371202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:02.371204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:02.371425 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:02.371570 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:02.371571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:02.371571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:02.371656 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:02.371882 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:02.371887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:02.372001 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 06:57:02.372081 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 06:57:02.372195 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260808 06:57:02.372824 22778 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46569
I20260808 06:57:02.372910 22778 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:02.373077 22778 heartbeater.cc:511] Master 127.21.28.126:46569 requested a full tablet report, sending...
I20260808 06:57:02.373328 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22588
I20260808 06:57:02.373407 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:02.373564 21617 external_mini_cluster.cc:1428] Setting key eb0b428eec24e99cf3909aeb8087dafc
I20260808 06:57:02.373598 22500 ts_manager.cc:194] Registered new tserver with Master: ec67db3437a94eb291f5bcafb7ac4e36 (127.21.28.65:44397)
I20260808 06:57:02.374514 22500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:54475
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:02.396775 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46569
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:02.476141 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:02.476346 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:02.476373 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:02.477483 22784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:02.477521 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:02.477548 22784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:02.478991 22784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46569
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:02.479207 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:02.479413 22784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:02.484053 22797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:02.484053 22800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:02.484055 22795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:02.484198 22784 server_base.cc:1034] running on GCE node
I20260808 06:57:02.484530 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:02.484716 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:02.485895 22784 hybrid_clock.cc:648] HybridClock initialized: now 1786172222485878 us; error 37 us; skew 500 ppm
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:02.488382 22784 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:02.488756 22784 webserver.cc:460] Webserver started at http://127.21.28.66:36479/ using document root <none> and password file <none>
I20260808 06:57:02.488967 22784 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:02.489022 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:02.489149 22784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:02.490145 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "dac3598e2ada41d5a2986bd803300d49"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "1615beed758f8a1117edb1708bfe9232"
server_key_iv: "bb204dc0d8edfdf202cd82ae60294b25"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.490468 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "dac3598e2ada41d5a2986bd803300d49"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "1615beed758f8a1117edb1708bfe9232"
server_key_iv: "bb204dc0d8edfdf202cd82ae60294b25"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.491691 22784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 06:57:02.492403 22814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.492578 22784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:02.492663 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "dac3598e2ada41d5a2986bd803300d49"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "1615beed758f8a1117edb1708bfe9232"
server_key_iv: "bb204dc0d8edfdf202cd82ae60294b25"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.492753 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:02.507423 22784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:02.509114 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:02.509236 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:02.509415 22784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:02.509717 22784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:02.509753 22784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.509778 22784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:02.509805 22784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.515983 22784 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:44511
I20260808 06:57:02.516021 22967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:44511 every 8 connection(s)
I20260808 06:57:02.516314 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:02.521046 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22784
I20260808 06:57:02.521130 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:02.521277 21617 external_mini_cluster.cc:1428] Setting key 3c3f94c75fa5a03b3dc79b5aa1d4b818
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:02.525916 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.517011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:49089 (local address 127.21.28.126:46569)
0808 06:57:02.517067 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.517069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.517241 (+ 172us) server_negotiation.cc:407] Connection header received
0808 06:57:02.518130 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.518132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.518144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.518191 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:02.518775 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.519260 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.520238 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.520439 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.520495 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.522834 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:02.522836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:02.522836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:02.522845 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:02.524441 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.524943 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.524944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.524944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.525004 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.525292 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.525292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.525293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.525424 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:02.525489 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.525734 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.525817 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:57:02.526044 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.516976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46569 (local address 127.21.28.66:49089)
0808 06:57:02.517128 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:02.517163 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:02.518023 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:02.518265 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:02.518269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:02.518511 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:02.518682 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.518686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.519339 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.519341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.520151 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.520153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.520507 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.520508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.520622 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.521043 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:02.521050 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:02.522644 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:02.524567 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:02.524573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:02.524575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:02.524862 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:02.525081 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:02.525082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:02.525082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:02.525221 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:02.525480 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:02.525487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:02.525618 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 06:57:02.525702 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 06:57:02.525842 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260808 06:57:02.526494 22968 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46569
I20260808 06:57:02.526575 22968 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:02.526731 22968 heartbeater.cc:511] Master 127.21.28.126:46569 requested a full tablet report, sending...
I20260808 06:57:02.527096 22500 ts_manager.cc:194] Registered new tserver with Master: dac3598e2ada41d5a2986bd803300d49 (127.21.28.66:44511)
I20260808 06:57:02.527588 22500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:49089
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:02.545626 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46569
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:02.624147 22973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:02.624323 22973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:02.624346 22973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:02.625418 22973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:02.625451 22973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:02.625475 22973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:02.626888 22973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46569
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:02.627051 22973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:02.627213 22973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:02.631645 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:02.631661 22980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:02.631716 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:02.631880 22973 server_base.cc:1034] running on GCE node
I20260808 06:57:02.632077 22973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:02.632280 22973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:02.633412 22973 hybrid_clock.cc:648] HybridClock initialized: now 1786172222633402 us; error 28 us; skew 500 ppm
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:02.635821 22973 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:02.636222 22973 webserver.cc:460] Webserver started at http://127.21.28.67:36813/ using document root <none> and password file <none>
I20260808 06:57:02.636415 22973 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:02.636454 22973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:02.636540 22973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:02.637449 22973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "d8d32a6917734787be0c12d3f5b1ea19"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "7a2dc671bf7684af36212e68bd673f30"
server_key_iv: "85a19b9202a52561fefddcafdf3e24a0"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.637724 22973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "d8d32a6917734787be0c12d3f5b1ea19"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "7a2dc671bf7684af36212e68bd673f30"
server_key_iv: "85a19b9202a52561fefddcafdf3e24a0"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.638854 22973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:02.639468 22994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.639608 22973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.639701 22973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "d8d32a6917734787be0c12d3f5b1ea19"
format_stamp: "Formatted at 2026-08-08 06:57:02 on dist-test-slave-nj21"
server_key: "7a2dc671bf7684af36212e68bd673f30"
server_key_iv: "85a19b9202a52561fefddcafdf3e24a0"
server_key_version: "encryptionkey@0"
I20260808 06:57:02.639766 22973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:02.649372 22973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:02.650904 22973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:02.651003 22973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:02.651217 22973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:02.651463 22973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:02.651492 22973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.651510 22973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:02.651527 22973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:02.657001 22973 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:46383
I20260808 06:57:02.657060 23156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:46383 every 8 connection(s)
I20260808 06:57:02.657296 22973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:02.659448 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22973
I20260808 06:57:02.659545 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:02.659740 21617 external_mini_cluster.cc:1428] Setting key 5007ec5b955cae851c0b0442974d151a
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:02.665721 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.657883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:36437 (local address 127.21.28.126:46569)
0808 06:57:02.657952 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.657954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.658102 (+ 148us) server_negotiation.cc:407] Connection header received
0808 06:57:02.658912 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.658913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.658925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.658991 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:02.659700 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.660305 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.661267 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.661475 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.661530 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.663424 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:02.663426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:02.663426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:02.663433 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:02.664522 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.664861 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.664872 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.664873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.664928 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.665133 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.665134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.665135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.665246 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:02.665306 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.665552 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.665632 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 06:57:02.665860 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.657841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46569 (local address 127.21.28.67:36437)
0808 06:57:02.657998 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:02.658030 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:02.658816 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:02.659038 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:02.659041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:02.659259 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:02.659541 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.659547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.660404 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.660405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.661183 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.661185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.661547 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.661548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.661665 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.662074 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:02.662081 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:02.663334 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:02.664592 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:02.664598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:02.664600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:02.664797 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:02.664981 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:02.664982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:02.664982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:02.665080 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:02.665302 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:02.665307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:02.665441 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 06:57:02.665523 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 06:57:02.665667 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260808 06:57:02.666255 23157 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46569
I20260808 06:57:02.666342 23157 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:02.666494 23157 heartbeater.cc:511] Master 127.21.28.126:46569 requested a full tablet report, sending...
I20260808 06:57:02.666854 22500 ts_manager.cc:194] Registered new tserver with Master: d8d32a6917734787be0c12d3f5b1ea19 (127.21.28.67:46383)
I20260808 06:57:02.667176 22500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:36437
I20260808 06:57:02.671167 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:57:02.678115 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.672946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.21.28.126:46569)
0808 06:57:02.673000 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.673002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.673097 (+ 95us) server_negotiation.cc:407] Connection header received
0808 06:57:02.673156 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.673156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.673166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.673206 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:02.673723 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.674171 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.674904 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.675093 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.675143 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.676164 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:02.676165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:02.676166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:02.676172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:02.677137 (+ 965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.677434 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.677435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.677436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.677483 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.677639 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.677640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.677640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.677770 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:02.677862 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.677963 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.678032 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 06:57:02.680591 22500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:02.682420 22500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:02.689219 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.684992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.21.28.67:46383)
0808 06:57:02.685152 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.685156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.685205 (+ 49us) server_negotiation.cc:407] Connection header received
0808 06:57:02.685244 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.685246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.685261 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.685350 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:02.685468 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.686533 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.688095 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.688791 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.688864 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.688920 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.689111 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.689147 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260808 06:57:02.689229 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.684789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:46383 (local address 127.0.0.1:59386)
0808 06:57:02.685140 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:02.685147 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:02.685201 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:02.685350 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:02.685352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:02.685354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:02.685413 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.685414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.686632 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.686633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.688020 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.688022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.688955 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.688956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.689010 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.689015 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:57:02.689069 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 06:57:02.689143 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":74,"thread_start_us":67,"threads_started":1}
I20260808 06:57:02.689338 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.685038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.21.28.66:44511)
0808 06:57:02.685238 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.685242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.685250 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:02.685288 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.685290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.685303 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.685393 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:02.685479 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.686543 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.688066 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.688791 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.688863 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.688920 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.689201 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.689253 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260808 06:57:02.689409 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.684789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:44511 (local address 127.0.0.1:45744)
0808 06:57:02.685036 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:02.685045 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:02.685083 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:02.685404 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:02.685405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:02.685407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:02.685436 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.685437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.686626 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.686627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.687967 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.687970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.688889 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.688890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.688968 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.688981 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 06:57:02.689063 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 06:57:02.689129 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":63,"thread_start_us":86,"threads_started":1}
I20260808 06:57:02.690129 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.684768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:44397 (local address 127.0.0.1:48382)
0808 06:57:02.684951 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:02.684985 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:02.685046 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:02.686844 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:02.686845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:02.686848 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:02.686923 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.686926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.687751 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.687752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.688562 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.688565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.689918 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.689919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.689980 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.689983 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:02.690029 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 06:57:02.690065 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":18,"thread_start_us":81,"threads_started":1}
I20260808 06:57:02.690075 23062 tablet_service.cc:1467] Processing CreateTablet for tablet b3e1ad26933d4445aa12982422c9af05 (DEFAULT_TABLE table=test-table [id=1885fb6363e0455f8df760ddf48f1b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:02.690152 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.684878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.21.28.65:44397)
0808 06:57:02.686707 (+ 1829us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.686711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.686718 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:02.686753 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.686754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.686766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.686828 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:02.686993 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.687653 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.688643 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.689840 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.689903 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.689959 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.690041 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.690082 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1791,"thread_start_us":77,"threads_started":1}
I20260808 06:57:02.690335 23062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e1ad26933d4445aa12982422c9af05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:02.692575 22874 tablet_service.cc:1467] Processing CreateTablet for tablet b3e1ad26933d4445aa12982422c9af05 (DEFAULT_TABLE table=test-table [id=1885fb6363e0455f8df760ddf48f1b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:02.692915 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e1ad26933d4445aa12982422c9af05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:02.693056 23173 tablet_bootstrap.cc:492] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19: Bootstrap starting.
I20260808 06:57:02.693406 22684 tablet_service.cc:1467] Processing CreateTablet for tablet b3e1ad26933d4445aa12982422c9af05 (DEFAULT_TABLE table=test-table [id=1885fb6363e0455f8df760ddf48f1b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:02.693727 23173 tablet_bootstrap.cc:654] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:02.693730 22684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e1ad26933d4445aa12982422c9af05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:02.694396 23173 log.cc:824] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:02.695047 23173 tablet_bootstrap.cc:492] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19: No bootstrap required, opened a new log
I20260808 06:57:02.695108 23173 ts_tablet_manager.cc:1397] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:02.696280 23175 tablet_bootstrap.cc:492] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49: Bootstrap starting.
I20260808 06:57:02.697280 23175 tablet_bootstrap.cc:654] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:02.698396 23175 log.cc:824] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:02.698308 23173 raft_consensus.cc:348] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.698539 23173 raft_consensus.cc:374] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:02.698588 23173 raft_consensus.cc:729] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d32a6917734787be0c12d3f5b1ea19, State: Initialized, Role: FOLLOWER
I20260808 06:57:02.698714 23173 consensus_queue.cc:260] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.699040 23173 ts_tablet_manager.cc:1428] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260808 06:57:02.699071 23157 heartbeater.cc:503] Master 127.21.28.126:46569 was elected leader, sending a full tablet report...
I20260808 06:57:02.699276 23175 tablet_bootstrap.cc:492] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49: No bootstrap required, opened a new log
I20260808 06:57:02.699280 23178 tablet_bootstrap.cc:492] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36: Bootstrap starting.
I20260808 06:57:02.699350 23175 ts_tablet_manager.cc:1397] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:57:02.700363 23178 tablet_bootstrap.cc:654] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:02.701087 23178 log.cc:824] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:02.701938 23178 tablet_bootstrap.cc:492] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36: No bootstrap required, opened a new log
I20260808 06:57:02.702011 23178 ts_tablet_manager.cc:1397] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 06:57:02.703421 23175 raft_consensus.cc:348] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.703603 23175 raft_consensus.cc:374] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:02.703647 23175 raft_consensus.cc:729] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac3598e2ada41d5a2986bd803300d49, State: Initialized, Role: FOLLOWER
I20260808 06:57:02.703776 23175 consensus_queue.cc:260] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.704082 23175 ts_tablet_manager.cc:1428] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 06:57:02.704145 22968 heartbeater.cc:503] Master 127.21.28.126:46569 was elected leader, sending a full tablet report...
I20260808 06:57:02.705224 23178 raft_consensus.cc:348] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.705384 23178 raft_consensus.cc:374] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:02.705410 23178 raft_consensus.cc:729] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec67db3437a94eb291f5bcafb7ac4e36, State: Initialized, Role: FOLLOWER
I20260808 06:57:02.705493 23178 consensus_queue.cc:260] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.705870 23178 ts_tablet_manager.cc:1428] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260808 06:57:02.705878 22778 heartbeater.cc:503] Master 127.21.28.126:46569 was elected leader, sending a full tablet report...
I20260808 06:57:02.707986 23183 raft_consensus.cc:482] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:02.708052 23183 raft_consensus.cc:504] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.708549 23183 leader_election.cc:288] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec67db3437a94eb291f5bcafb7ac4e36 (127.21.28.65:44397), d8d32a6917734787be0c12d3f5b1ea19 (127.21.28.67:46383)
I20260808 06:57:02.712292 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.708721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:53965 (local address 127.21.28.65:44397)
0808 06:57:02.708776 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.708778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.708825 (+ 47us) server_negotiation.cc:407] Connection header received
0808 06:57:02.708898 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.708899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.708908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.708962 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:02.709155 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.709880 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.711245 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.711928 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.712001 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.712010 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.712202 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.712234 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:57:02.712311 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.708741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:51491 (local address 127.21.28.67:46383)
0808 06:57:02.708778 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.708779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.708895 (+ 116us) server_negotiation.cc:407] Connection header received
0808 06:57:02.708935 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.708936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.708944 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.708990 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:02.709154 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.709886 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.711316 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.711961 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.712020 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.712027 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.712202 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.712236 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:02.712361 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.708678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:46383 (local address 127.21.28.66:51491)
0808 06:57:02.708831 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:02.708840 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:02.708890 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:02.708989 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:02.708989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:02.708990 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:02.709089 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.709091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.709958 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.709958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.711244 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.711246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.712020 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.712021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.712131 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.712135 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:02.712193 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 06:57:02.712241 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":40,"thread_start_us":71,"threads_started":1}
I20260808 06:57:02.712538 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.708669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:44397 (local address 127.21.28.66:53965)
0808 06:57:02.708771 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:02.708781 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:02.708842 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:02.708963 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:02.708964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:02.708966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:02.709091 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.709092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.709958 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.709958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.711165 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:02.711168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.711996 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.711997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:02.712131 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.712135 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:02.712193 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 06:57:02.712241 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":9}
I20260808 06:57:02.712742 23088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e1ad26933d4445aa12982422c9af05" candidate_uuid: "dac3598e2ada41d5a2986bd803300d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d32a6917734787be0c12d3f5b1ea19" is_pre_election: true
I20260808 06:57:02.712836 22711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e1ad26933d4445aa12982422c9af05" candidate_uuid: "dac3598e2ada41d5a2986bd803300d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" is_pre_election: true
I20260808 06:57:02.712932 23088 raft_consensus.cc:2454] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dac3598e2ada41d5a2986bd803300d49 in term 0.
I20260808 06:57:02.712960 22711 raft_consensus.cc:2454] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dac3598e2ada41d5a2986bd803300d49 in term 0.
I20260808 06:57:02.713230 22819 leader_election.cc:302] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dac3598e2ada41d5a2986bd803300d49, ec67db3437a94eb291f5bcafb7ac4e36; no voters:
I20260808 06:57:02.713364 23183 raft_consensus.cc:2781] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:02.713415 23183 raft_consensus.cc:482] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:02.713430 23183 raft_consensus.cc:3037] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:02.714368 23183 raft_consensus.cc:504] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.714478 23183 leader_election.cc:288] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [CANDIDATE]: Term 1 election: Requested vote from peers ec67db3437a94eb291f5bcafb7ac4e36 (127.21.28.65:44397), d8d32a6917734787be0c12d3f5b1ea19 (127.21.28.67:46383)
I20260808 06:57:02.714609 23088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e1ad26933d4445aa12982422c9af05" candidate_uuid: "dac3598e2ada41d5a2986bd803300d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d32a6917734787be0c12d3f5b1ea19"
I20260808 06:57:02.714671 23088 raft_consensus.cc:3037] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:02.714758 22711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e1ad26933d4445aa12982422c9af05" candidate_uuid: "dac3598e2ada41d5a2986bd803300d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec67db3437a94eb291f5bcafb7ac4e36"
I20260808 06:57:02.714829 22711 raft_consensus.cc:3037] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:02.715587 23088 raft_consensus.cc:2454] T b3e1ad26933d4445aa12982422c9af05 P d8d32a6917734787be0c12d3f5b1ea19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac3598e2ada41d5a2986bd803300d49 in term 1.
I20260808 06:57:02.715824 22815 leader_election.cc:302] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8d32a6917734787be0c12d3f5b1ea19, dac3598e2ada41d5a2986bd803300d49; no voters:
I20260808 06:57:02.715893 22711 raft_consensus.cc:2454] T b3e1ad26933d4445aa12982422c9af05 P ec67db3437a94eb291f5bcafb7ac4e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac3598e2ada41d5a2986bd803300d49 in term 1.
I20260808 06:57:02.715911 23183 raft_consensus.cc:2781] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:02.716043 23183 raft_consensus.cc:686] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [term 1 LEADER]: Becoming Leader. State: Replica: dac3598e2ada41d5a2986bd803300d49, State: Running, Role: LEADER
I20260808 06:57:02.716131 23183 consensus_queue.cc:237] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } }
I20260808 06:57:02.716854 22500 catalog_manager.cc:5495] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49 reported cstate change: term changed from 0 to 1, leader changed from <none> to dac3598e2ada41d5a2986bd803300d49 (127.21.28.66). New cstate: current_term: 1 leader_uuid: "dac3598e2ada41d5a2986bd803300d49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec67db3437a94eb291f5bcafb7ac4e36" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dac3598e2ada41d5a2986bd803300d49" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d32a6917734787be0c12d3f5b1ea19" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 46383 } health_report { overall_health: UNKNOWN } } }
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:02.728025 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.721013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.21.28.65:44397)
0808 06:57:02.721072 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.721074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.721183 (+ 109us) server_negotiation.cc:407] Connection header received
0808 06:57:02.721232 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.721233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.721241 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.721271 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:02.721804 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.722227 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.722873 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.723075 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.723135 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.725696 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:02.725703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:02.725704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:02.725710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:02.727120 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.727418 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.727420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.727420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.727469 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.727597 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.727598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.727598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.727745 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:02.727846 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.727908 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.727967 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:02.750542 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.744115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.21.28.65:44397)
0808 06:57:02.744169 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.744172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.744227 (+ 55us) server_negotiation.cc:407] Connection header received
0808 06:57:02.744297 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.744298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.744308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.744355 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:02.744889 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.745465 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.746154 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.746399 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.746457 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.748523 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:02.748524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:02.748524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:02.748530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:02.749753 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.750015 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.750016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.750016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.750068 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.750182 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.750183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.750183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.750258 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:02.750319 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.750425 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.750477 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260808 06:57:02.766902 22969 tablet.cc:2369] T b3e1ad26933d4445aa12982422c9af05 P dac3598e2ada41d5a2986bd803300d49: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[22435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:02.772042 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:02.765778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.21.28.65:44397)
0808 06:57:02.765842 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:02.765843 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:02.765902 (+ 59us) server_negotiation.cc:407] Connection header received
0808 06:57:02.765956 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:02.765957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:02.765965 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:02.766010 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:02.766461 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.766996 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.767741 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:02.767945 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:02.767998 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:02.770063 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:02.770064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:02.770064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:02.770070 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:02.771129 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.771426 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.771427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.771427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.771495 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:02.771643 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:02.771643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:02.771644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:02.771770 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:02.771843 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:02.771921 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 06:57:02.771975 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 06:57:02.772759 22655 tablet_service.cc:2908] Rejecting scan request for tablet b3e1ad26933d4445aa12982422c9af05: Uninitialized: clean time has not yet been initialized
I20260808 06:57:02.773511 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22588
I20260808 06:57:02.789557 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22784
I20260808 06:57:02.806443 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22973
I20260808 06:57:02.814267 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22452
2026-08-08T06:57:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1965 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[23203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[23203](info): set up 2 sockets
Aug 08 06:57:02 dist-test-slave-nj21 krb5kdc[23203](info): commencing operation
krb5kdc: starting...
W20260808 06:57:03.947521 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:04.057719 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36905
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:04.140265 23222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:04.140471 23222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:04.140499 23222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:04.141636 23222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:04.141680 23222 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 06:57:04.141698 23222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:04.141716 23222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:04.141731 23222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:04.141747 23222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:04.143247 23222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36905
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:04.143487 23222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:04.143709 23222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:04.148307 23231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:04.148308 23228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:04.148348 23222 server_base.cc:1034] running on GCE node
W20260808 06:57:04.148309 23227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:04.148711 23222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:04.148985 23222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:04.150130 23222 hybrid_clock.cc:648] HybridClock initialized: now 1786172224150112 us; error 35 us; skew 500 ppm
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:04.156766 23222 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:04.157171 23222 webserver.cc:460] Webserver started at http://127.21.28.126:39221/ using document root <none> and password file <none>
I20260808 06:57:04.157392 23222 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:04.157440 23222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:04.157563 23222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:04.158573 23222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "706c7277b7c6404aa595eccbf67f9ec8"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "df942808f83c80b5b305862959b7a21a"
server_key_iv: "1ff70a704a3e40f8be7a5c371b0763de"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.158898 23222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "706c7277b7c6404aa595eccbf67f9ec8"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "df942808f83c80b5b305862959b7a21a"
server_key_iv: "1ff70a704a3e40f8be7a5c371b0763de"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.160178 23222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:04.160815 23241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:04.160944 23222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:04.161029 23222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "706c7277b7c6404aa595eccbf67f9ec8"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "df942808f83c80b5b305862959b7a21a"
server_key_iv: "1ff70a704a3e40f8be7a5c371b0763de"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.161085 23222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:04.176906 23222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:04.178431 23222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:04.178555 23222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:04.183101 23222 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:36905
I20260808 06:57:04.183143 23334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:36905 every 8 connection(s)
I20260808 06:57:04.183440 23222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:04.184070 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.186213 23335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8: Bootstrap starting.
I20260808 06:57:04.186757 23335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.187008 23335 log.cc:824] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:04.187664 23335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8: No bootstrap required, opened a new log
I20260808 06:57:04.189826 23335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706c7277b7c6404aa595eccbf67f9ec8" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36905 } }
I20260808 06:57:04.189955 23335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.189986 23335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706c7277b7c6404aa595eccbf67f9ec8, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.190111 23335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706c7277b7c6404aa595eccbf67f9ec8" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36905 } }
I20260808 06:57:04.190171 23335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:04.190199 23335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:04.190245 23335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:04.190734 23335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706c7277b7c6404aa595eccbf67f9ec8" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36905 } }
I20260808 06:57:04.190827 23335 leader_election.cc:302] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 706c7277b7c6404aa595eccbf67f9ec8; no voters:
I20260808 06:57:04.190974 23335 leader_election.cc:288] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:04.191015 23338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:04.191184 23338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [term 1 LEADER]: Becoming Leader. State: Replica: 706c7277b7c6404aa595eccbf67f9ec8, State: Running, Role: LEADER
I20260808 06:57:04.191213 23335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:04.191294 23338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706c7277b7c6404aa595eccbf67f9ec8" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36905 } }
I20260808 06:57:04.191690 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "706c7277b7c6404aa595eccbf67f9ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706c7277b7c6404aa595eccbf67f9ec8" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36905 } } }
I20260808 06:57:04.191705 23340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 706c7277b7c6404aa595eccbf67f9ec8. Latest consensus state: current_term: 1 leader_uuid: "706c7277b7c6404aa595eccbf67f9ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "706c7277b7c6404aa595eccbf67f9ec8" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36905 } } }
I20260808 06:57:04.191788 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:04.191805 23340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:04.192000 23352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:04.192013 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23222
I20260808 06:57:04.192095 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:04.192260 21617 external_mini_cluster.cc:1428] Setting key f5be0222d216aa9f992fac03739d8830
I20260808 06:57:04.192505 23352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:04.193957 23352 catalog_manager.cc:1347] Generated new cluster ID: ecd8378b325e4f8dbb73a7253bfa61f4
I20260808 06:57:04.194010 23352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:04.203038 23352 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:04.203531 23352 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:04.204289 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.192719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.21.28.126:36905)
0808 06:57:04.193339 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.193349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.193391 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:57:04.193510 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.193517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.193570 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.193704 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.194439 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.195777 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.196861 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.197484 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.197893 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.200403 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.200407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.200407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.200419 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.202296 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.202707 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.202710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.202710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.202809 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.203035 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.203036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.203036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.203339 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.203852 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.203932 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.204078 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":56,"threads_started":1}
I20260808 06:57:04.214378 23352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8: Generated new TSK 0
I20260808 06:57:04.214537 23352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:04.243594 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36905
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:04.323777 23371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:04.323973 23371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:04.324004 23371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:04.325132 23371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:04.325171 23371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:04.325198 23371 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:04.325222 23371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:04.326632 23371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:04.326879 23371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:04.327077 23371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:04.332437 23381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:04.332444 23376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:04.332465 23377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:04.332823 23371 server_base.cc:1034] running on GCE node
I20260808 06:57:04.333012 23371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:04.333227 23371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:04.334367 23371 hybrid_clock.cc:648] HybridClock initialized: now 1786172224334355 us; error 24 us; skew 500 ppm
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:04.336740 23371 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:04.337179 23371 webserver.cc:460] Webserver started at http://127.21.28.65:39995/ using document root <none> and password file <none>
I20260808 06:57:04.337393 23371 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:04.337442 23371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:04.337565 23371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:04.338560 23371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "93533afcf59147849cd34cac198236b9"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "4eeb6a88a8311d1066e89eaa6559acfa"
server_key_iv: "85873adf39244ab65a35e8920377af32"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.338893 23371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "93533afcf59147849cd34cac198236b9"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "4eeb6a88a8311d1066e89eaa6559acfa"
server_key_iv: "85873adf39244ab65a35e8920377af32"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.340191 23371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:04.340845 23390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:04.341003 23371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:04.341089 23371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "93533afcf59147849cd34cac198236b9"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "4eeb6a88a8311d1066e89eaa6559acfa"
server_key_iv: "85873adf39244ab65a35e8920377af32"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.341144 23371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:04.348371 23371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:04.349956 23371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:04.350078 23371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:04.350538 23371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:04.350576 23371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:04.350607 23371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:04.350636 23371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:04.357831 23371 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:41305
I20260808 06:57:04.357908 23552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:41305 every 8 connection(s)
I20260808 06:57:04.358177 23371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:04.359261 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.351012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:35135 (local address 127.21.28.126:36905)
0808 06:57:04.351074 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.351077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.351235 (+ 158us) server_negotiation.cc:407] Connection header received
0808 06:57:04.351660 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.351661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.351703 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.351777 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.352346 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.352875 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.353837 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.354046 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.354100 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.356648 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.356650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.356650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.356658 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.358046 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.358417 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.358419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.358419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.358493 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.358752 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.358754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.358754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.358880 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.358931 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.359122 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.359197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:57:04.359418 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.350962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36905 (local address 127.21.28.65:35135)
0808 06:57:04.351114 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.351152 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.351574 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.351830 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.351833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.352055 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:04.352259 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.352263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.352958 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.352959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.353758 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.353761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.354114 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.354115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.354244 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.354732 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:04.354742 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:04.356564 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:04.358112 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.358116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.358118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.358336 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.358580 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.358581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.358582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.358704 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.358929 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:04.358932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:04.359037 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.359108 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.359223 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260808 06:57:04.359934 23553 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36905
I20260808 06:57:04.360013 23553 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:04.360169 23553 heartbeater.cc:511] Master 127.21.28.126:36905 requested a full tablet report, sending...
I20260808 06:57:04.360684 23266 ts_manager.cc:194] Registered new tserver with Master: 93533afcf59147849cd34cac198236b9 (127.21.28.65:41305)
I20260808 06:57:04.361567 23266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:35135
I20260808 06:57:04.365789 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.361017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:59261 (local address 127.21.28.126:36905)
0808 06:57:04.361052 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.361054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.361113 (+ 59us) server_negotiation.cc:407] Connection header received
0808 06:57:04.361177 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.361177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.361185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.361224 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.361539 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.362168 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.362825 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.363049 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.363102 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.363986 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.363987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.363987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.363992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.365039 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.365306 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.365307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.365307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.365347 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.365467 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.365467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.365468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.365547 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.365587 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.365685 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.365734 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 06:57:04.365810 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.360934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36905 (local address 127.21.28.65:59261)
0808 06:57:04.361052 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.361062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.361127 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.361229 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.361229 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.361415 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:04.361478 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.361480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.362248 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.362248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.362737 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.362738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.363103 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.363103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.363155 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.363528 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:04.363533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:04.363923 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:04.365100 (+ 1177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.365101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.365101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.365253 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.365389 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.365390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.365390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.365435 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.365584 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:04.365585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:04.365627 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.365670 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.365737 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260808 06:57:04.368371 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23371
I20260808 06:57:04.368449 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:04.368590 21617 external_mini_cluster.cc:1428] Setting key 64c140a2821b373a4cc2b4804f7386d0
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:04.392107 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36905
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:04.471069 23563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:04.471257 23563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:04.471279 23563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:04.472419 23563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:04.472460 23563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:04.472478 23563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:04.472496 23563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:04.473889 23563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:04.474081 23563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:04.474278 23563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:57:04.478953 23563 server_base.cc:1034] running on GCE node
W20260808 06:57:04.478905 23570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:04.478902 23569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:04.478916 23572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:04.479271 23563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:04.479492 23563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:04.480625 23563 hybrid_clock.cc:648] HybridClock initialized: now 1786172224480601 us; error 39 us; skew 500 ppm
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:04.482941 23563 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:04.483330 23563 webserver.cc:460] Webserver started at http://127.21.28.66:39507/ using document root <none> and password file <none>
I20260808 06:57:04.483525 23563 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:04.483570 23563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:04.483724 23563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:04.484702 23563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "ff77c9e5e7ac4dedaae105557e0470c9"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "1ab96695f6bf455b8f475ed6c4e0ebd8"
server_key_iv: "b84013743d6a523b0badd679f278cd70"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.485021 23563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "ff77c9e5e7ac4dedaae105557e0470c9"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "1ab96695f6bf455b8f475ed6c4e0ebd8"
server_key_iv: "b84013743d6a523b0badd679f278cd70"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.486188 23563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:04.486817 23582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:04.486985 23563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:04.487067 23563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "ff77c9e5e7ac4dedaae105557e0470c9"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "1ab96695f6bf455b8f475ed6c4e0ebd8"
server_key_iv: "b84013743d6a523b0badd679f278cd70"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.487129 23563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:04.501299 23563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:04.502837 23563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:04.502954 23563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:04.503417 23563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:04.503453 23563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:04.503481 23563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:04.503496 23563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:04.510671 23563 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:46425
I20260808 06:57:04.510810 23753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:46425 every 8 connection(s)
I20260808 06:57:04.511050 23563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:04.512909 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.503759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:51925 (local address 127.21.28.126:36905)
0808 06:57:04.503832 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.503835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.504082 (+ 247us) server_negotiation.cc:407] Connection header received
0808 06:57:04.504657 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.504658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.504669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.504723 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.505563 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.506353 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.507823 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.508113 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.508187 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.510472 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.510474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.510474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.510482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.511906 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.512240 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.512240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.512241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.512286 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.512442 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.512443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.512443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.512539 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.512583 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.512777 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.512835 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 06:57:04.513077 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.503763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36905 (local address 127.21.28.66:51925)
0808 06:57:04.503944 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.503991 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.504562 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.504812 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.504817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.505169 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:04.505444 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.505451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.506453 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.506454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.507739 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.507743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.508205 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.508207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.508351 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.508856 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:04.508874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:04.510388 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:04.511980 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.511985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.511986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.512188 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.512321 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.512321 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.512322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.512404 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.512581 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:04.512584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:04.512685 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.512758 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.512884 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260808 06:57:04.513511 23754 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36905
I20260808 06:57:04.513586 23754 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:04.513738 23754 heartbeater.cc:511] Master 127.21.28.126:36905 requested a full tablet report, sending...
I20260808 06:57:04.514112 23265 ts_manager.cc:194] Registered new tserver with Master: ff77c9e5e7ac4dedaae105557e0470c9 (127.21.28.66:46425)
I20260808 06:57:04.514554 23265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:51925
I20260808 06:57:04.516197 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23563
I20260808 06:57:04.516283 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:04.516454 21617 external_mini_cluster.cc:1428] Setting key 30934cbfdc956f71a56d74fceecac1f2
I20260808 06:57:04.519578 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.514486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:40161 (local address 127.21.28.126:36905)
0808 06:57:04.514519 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.514520 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.514660 (+ 140us) server_negotiation.cc:407] Connection header received
0808 06:57:04.514679 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.514680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.514688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.514733 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.515022 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.515465 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.516233 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.516462 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.516522 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.517422 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.517423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.517424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.517429 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.518587 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.518876 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.518877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.518877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.518927 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.519084 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.519085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.519085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.519218 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.519274 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.519426 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.519490 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 06:57:04.519593 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.514434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36905 (local address 127.21.28.66:40161)
0808 06:57:04.514560 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.514569 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.514622 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.514747 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.514748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.514901 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:04.514953 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.514955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.515549 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.515549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.516157 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.516160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.516550 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.516551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.516630 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.517007 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:04.517011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:04.517340 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:04.518659 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.518660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.518661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.518816 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.518979 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.518980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.518980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.519035 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.519276 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:04.519277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:04.519320 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.519396 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.519491 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:04.540419 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36905
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:04.630746 23768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:04.630946 23768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:04.630968 23768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:04.632112 23768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:04.632150 23768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:04.632170 23768 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:04.632200 23768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:04.633579 23768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:04.633785 23768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:04.633993 23768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:04.638664 23779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:04.638636 23774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:04.638674 23768 server_base.cc:1034] running on GCE node
W20260808 06:57:04.638657 23775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:04.638959 23768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:04.639169 23768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:04.640343 23768 hybrid_clock.cc:648] HybridClock initialized: now 1786172224640330 us; error 32 us; skew 500 ppm
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:04.642685 23768 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:04.643092 23768 webserver.cc:460] Webserver started at http://127.21.28.67:42289/ using document root <none> and password file <none>
I20260808 06:57:04.643277 23768 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:04.643322 23768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:04.643443 23768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:04.644516 23768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "2bc04e5fb05348c6ada90953e4180a0a"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "83bde1aeae50aeb479897ed5d645686c"
server_key_iv: "8bd8153e2befc05b94c147379602c72c"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.644866 23768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "2bc04e5fb05348c6ada90953e4180a0a"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "83bde1aeae50aeb479897ed5d645686c"
server_key_iv: "8bd8153e2befc05b94c147379602c72c"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.646039 23768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 06:57:04.646658 23788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:04.646802 23768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:04.646895 23768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "2bc04e5fb05348c6ada90953e4180a0a"
format_stamp: "Formatted at 2026-08-08 06:57:04 on dist-test-slave-nj21"
server_key: "83bde1aeae50aeb479897ed5d645686c"
server_key_iv: "8bd8153e2befc05b94c147379602c72c"
server_key_version: "encryptionkey@0"
I20260808 06:57:04.646960 23768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:04.676643 23768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:04.678651 23768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:04.678759 23768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:04.679210 23768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:04.679250 23768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:04.679279 23768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:04.679297 23768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:04.686828 23768 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:39229
I20260808 06:57:04.686869 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:39229 every 8 connection(s)
I20260808 06:57:04.687268 23768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:04.688887 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.679690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:42089 (local address 127.21.28.126:36905)
0808 06:57:04.679743 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.679745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.679934 (+ 189us) server_negotiation.cc:407] Connection header received
0808 06:57:04.680389 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.680391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.680403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.680484 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.681084 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.681811 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.683278 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.683594 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.683702 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.686078 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.686079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.686080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.686089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.687754 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.688105 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.688106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.688106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.688167 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.688393 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.688394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.688394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.688515 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.688576 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.688765 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.688826 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:04.689035 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.679625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36905 (local address 127.21.28.67:42089)
0808 06:57:04.679801 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.679839 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.680305 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.680544 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.680547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.680769 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:04.681010 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.681016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.681915 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.681916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.683159 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.683162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.683711 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.683713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.683818 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.684281 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:04.684294 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:04.685997 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:04.687839 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.687841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.687843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.688041 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.688233 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.688233 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.688234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.688317 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.688576 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:04.688578 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:04.688677 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.688745 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.688853 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":109,"threads_started":1}
I20260808 06:57:04.689494 23957 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36905
I20260808 06:57:04.689594 23957 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:04.689803 23957 heartbeater.cc:511] Master 127.21.28.126:36905 requested a full tablet report, sending...
I20260808 06:57:04.690148 23266 ts_manager.cc:194] Registered new tserver with Master: 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229)
I20260808 06:57:04.690613 23266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:42089
I20260808 06:57:04.695308 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23768
I20260808 06:57:04.695398 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:04.695438 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.690488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:52603 (local address 127.21.28.126:36905)
0808 06:57:04.690754 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.690756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.690764 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:04.690778 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.690778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.690785 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.690823 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.691154 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.691600 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.692350 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.692543 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.692593 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.693389 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.693390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.693390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.693396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.694441 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.694721 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.694722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.694723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.694774 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.694985 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.694986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.694986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.695088 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.695143 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.695290 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.695353 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260808 06:57:04.695461 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.690446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36905 (local address 127.21.28.67:52603)
0808 06:57:04.690570 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.690579 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.690622 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.690839 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.690840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.691028 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:04.691089 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.691091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.691701 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.691701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.692283 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.692285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.692599 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.692600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.692675 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.693042 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:04.693046 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:04.693296 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:04.694510 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.694511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.694511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.694664 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.694846 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.694847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.694847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.694903 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.695161 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:04.695162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:04.695205 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.695261 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.695353 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260808 06:57:04.695591 21617 external_mini_cluster.cc:1428] Setting key a997cb84847a849e53a354fffc6f4246
I20260808 06:57:04.696393 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:04.718377 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.712649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.21.28.126:36905)
0808 06:57:04.712706 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.712707 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.712716 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:04.712757 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.712758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.712766 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.712809 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.713264 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.713744 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.714494 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.714670 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.714720 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.716617 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.716618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.716618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.716624 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.717591 (+ 967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.717878 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.717878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.717878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.717918 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.718038 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.718038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.718039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.718120 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.718182 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.718265 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.718316 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:57:04.719862 23265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:04.721343 23265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:04.727936 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.723307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41305 (local address 127.0.0.1:47890)
0808 06:57:04.723601 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.723643 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.723729 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.723994 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.723997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.724000 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:04.724069 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.724070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.725398 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.725399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.726729 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.726732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.727540 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.727541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.727619 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.727633 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.727790 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.727853 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":27,"thread_start_us":47,"threads_started":1}
I20260808 06:57:04.728060 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.723307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:39229 (local address 127.0.0.1:41832)
0808 06:57:04.723623 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.723652 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.723729 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.724001 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.724002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.724003 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:04.724065 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.724067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.726208 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.726209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.727088 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.727091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.727798 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.727799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.727881 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.727885 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.727947 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.727987 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260808 06:57:04.728336 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.723427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.21.28.65:41305)
0808 06:57:04.723840 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.723844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.723855 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:57:04.723894 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.723896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.723911 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.723988 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:04.724131 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.725331 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.726816 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.727466 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.727519 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.727558 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.728191 (+ 633us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.728237 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":77,"threads_started":1}
I20260808 06:57:04.728380 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.723485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.21.28.67:39229)
0808 06:57:04.723870 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.723873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.723882 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:04.723912 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.723914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.723928 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.724000 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:04.725434 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.726120 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.727167 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.727737 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.727797 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.727849 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.727962 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.728289 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":80,"threads_started":1}
I20260808 06:57:04.728793 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.723486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46425 (local address 127.0.0.1:45950)
0808 06:57:04.724080 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.724089 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.724137 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.724327 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.724328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.724330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:04.724367 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.724368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.725508 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.725509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.726729 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.726732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.728356 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.728357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.728623 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.728629 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.728689 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.728731 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":19,"thread_start_us":75,"threads_started":1}
I20260808 06:57:04.728845 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.723596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.21.28.66:46425)
0808 06:57:04.723790 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.723794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.724190 (+ 396us) server_negotiation.cc:407] Connection header received
0808 06:57:04.724245 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.724247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.724262 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.724340 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:04.724426 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.725427 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.727588 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.728284 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.728353 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.728530 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.728712 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.728752 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
I20260808 06:57:04.729056 23464 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd65b51862743b7b5955ebacdd521b1 (DEFAULT_TABLE table=test-table [id=9d502701342c4b84b53ec319b02d2e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:04.729394 23464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd65b51862743b7b5955ebacdd521b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.732216 23988 tablet_bootstrap.cc:492] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Bootstrap starting.
I20260808 06:57:04.732259 23858 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd65b51862743b7b5955ebacdd521b1 (DEFAULT_TABLE table=test-table [id=9d502701342c4b84b53ec319b02d2e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:04.732460 23668 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd65b51862743b7b5955ebacdd521b1 (DEFAULT_TABLE table=test-table [id=9d502701342c4b84b53ec319b02d2e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:04.732611 23858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd65b51862743b7b5955ebacdd521b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.732795 23668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd65b51862743b7b5955ebacdd521b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.732832 23988 tablet_bootstrap.cc:654] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.733100 23988 log.cc:824] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:04.733719 23988 tablet_bootstrap.cc:492] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: No bootstrap required, opened a new log
I20260808 06:57:04.733777 23988 ts_tablet_manager.cc:1397] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:04.735301 23988 raft_consensus.cc:348] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.735435 23988 raft_consensus.cc:374] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.735469 23988 raft_consensus.cc:729] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93533afcf59147849cd34cac198236b9, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.735602 23988 consensus_queue.cc:260] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.735878 23988 ts_tablet_manager.cc:1428] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 06:57:04.735908 23990 tablet_bootstrap.cc:492] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: Bootstrap starting.
I20260808 06:57:04.736019 23553 heartbeater.cc:503] Master 127.21.28.126:36905 was elected leader, sending a full tablet report...
I20260808 06:57:04.736155 23996 tablet_bootstrap.cc:492] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Bootstrap starting.
I20260808 06:57:04.736806 23996 tablet_bootstrap.cc:654] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.737121 23996 log.cc:824] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:04.737154 23990 tablet_bootstrap.cc:654] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.737460 23990 log.cc:824] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:04.737818 23996 tablet_bootstrap.cc:492] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: No bootstrap required, opened a new log
I20260808 06:57:04.737885 23996 ts_tablet_manager.cc:1397] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:04.738180 23990 tablet_bootstrap.cc:492] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: No bootstrap required, opened a new log
I20260808 06:57:04.738246 23990 ts_tablet_manager.cc:1397] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:04.739290 23996 raft_consensus.cc:348] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.739413 23996 raft_consensus.cc:374] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.739442 23996 raft_consensus.cc:729] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff77c9e5e7ac4dedaae105557e0470c9, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.739549 23996 consensus_queue.cc:260] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.739615 23990 raft_consensus.cc:348] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.739789 23990 raft_consensus.cc:374] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.739836 23990 raft_consensus.cc:729] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc04e5fb05348c6ada90953e4180a0a, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.739848 23996 ts_tablet_manager.cc:1428] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 06:57:04.739882 23754 heartbeater.cc:503] Master 127.21.28.126:36905 was elected leader, sending a full tablet report...
I20260808 06:57:04.739972 23990 consensus_queue.cc:260] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.740224 23990 ts_tablet_manager.cc:1428] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:04.740247 23957 heartbeater.cc:503] Master 127.21.28.126:36905 was elected leader, sending a full tablet report...
W20260808 06:57:04.761719 23755 tablet.cc:2369] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:57:04.859246 23554 tablet.cc:2369] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:04.869184 23993 raft_consensus.cc:482] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:04.869302 23993 raft_consensus.cc:504] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.869663 23993 leader_election.cc:288] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229), ff77c9e5e7ac4dedaae105557e0470c9 (127.21.28.66:46425)
I20260808 06:57:04.873436 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.869976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:55949 (local address 127.21.28.67:39229)
0808 06:57:04.870019 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.870021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.870029 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:04.870090 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.870091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.870100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.870152 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:04.870312 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.871060 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.872521 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.873076 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.873145 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.873151 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.873327 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.873362 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:04.873436 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.869937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:42211 (local address 127.21.28.66:46425)
0808 06:57:04.869985 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.869987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.870028 (+ 41us) server_negotiation.cc:407] Connection header received
0808 06:57:04.870101 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.870102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.870110 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.870157 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:04.870311 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.871082 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.872521 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.873076 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.873143 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.873151 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.873327 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.873370 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 06:57:04.873466 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.869843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46425 (local address 127.21.28.65:42211)
0808 06:57:04.869966 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.869977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.870034 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.870166 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.870167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.870169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:04.870252 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.870254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.871139 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.871140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.872445 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.872450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.873163 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.873163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.873257 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.873265 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.873317 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.873365 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":58,"thread_start_us":62,"threads_started":1}
I20260808 06:57:04.873471 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.869864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:39229 (local address 127.21.28.65:55949)
0808 06:57:04.869971 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.869979 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.870034 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.870166 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.870166 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.870172 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:04.870251 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.870253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.871135 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.871136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.872446 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.872450 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.873161 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.873163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.873257 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.873267 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.873317 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.873365 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":80,"thread_start_us":59,"threads_started":1}
I20260808 06:57:04.873822 23695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd65b51862743b7b5955ebacdd521b1" candidate_uuid: "93533afcf59147849cd34cac198236b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" is_pre_election: true
I20260808 06:57:04.873822 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd65b51862743b7b5955ebacdd521b1" candidate_uuid: "93533afcf59147849cd34cac198236b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc04e5fb05348c6ada90953e4180a0a" is_pre_election: true
I20260808 06:57:04.873984 23891 raft_consensus.cc:2454] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93533afcf59147849cd34cac198236b9 in term 0.
I20260808 06:57:04.873984 23695 raft_consensus.cc:2454] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93533afcf59147849cd34cac198236b9 in term 0.
I20260808 06:57:04.874266 23396 leader_election.cc:302] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc04e5fb05348c6ada90953e4180a0a, 93533afcf59147849cd34cac198236b9; no voters:
I20260808 06:57:04.874411 23993 raft_consensus.cc:2781] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:04.874473 23993 raft_consensus.cc:482] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:04.874495 23993 raft_consensus.cc:3037] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:04.875198 23993 raft_consensus.cc:504] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.875320 23993 leader_election.cc:288] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [CANDIDATE]: Term 1 election: Requested vote from peers 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229), ff77c9e5e7ac4dedaae105557e0470c9 (127.21.28.66:46425)
I20260808 06:57:04.875497 23695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd65b51862743b7b5955ebacdd521b1" candidate_uuid: "93533afcf59147849cd34cac198236b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff77c9e5e7ac4dedaae105557e0470c9"
I20260808 06:57:04.875497 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd65b51862743b7b5955ebacdd521b1" candidate_uuid: "93533afcf59147849cd34cac198236b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc04e5fb05348c6ada90953e4180a0a"
I20260808 06:57:04.875582 23891 raft_consensus.cc:3037] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:04.875581 23695 raft_consensus.cc:3037] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:04.876492 23695 raft_consensus.cc:2454] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93533afcf59147849cd34cac198236b9 in term 1.
I20260808 06:57:04.876492 23891 raft_consensus.cc:2454] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93533afcf59147849cd34cac198236b9 in term 1.
I20260808 06:57:04.876646 23396 leader_election.cc:302] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc04e5fb05348c6ada90953e4180a0a, 93533afcf59147849cd34cac198236b9; no voters:
I20260808 06:57:04.876734 23993 raft_consensus.cc:2781] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:04.876842 23993 raft_consensus.cc:686] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 1 LEADER]: Becoming Leader. State: Replica: 93533afcf59147849cd34cac198236b9, State: Running, Role: LEADER
I20260808 06:57:04.876926 23993 consensus_queue.cc:237] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } }
I20260808 06:57:04.877672 23266 catalog_manager.cc:5495] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93533afcf59147849cd34cac198236b9 (127.21.28.65). New cstate: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } health_report { overall_health: UNKNOWN } } }
I20260808 06:57:04.910785 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.908585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.21.28.65:41305)
0808 06:57:04.908642 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.908644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.908669 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:57:04.908768 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.908769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.908786 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.908842 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:04.908943 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.909399 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.910128 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.910320 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.910370 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.910536 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:04.910631 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:04.910695 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.910713 (+ 18us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.910731 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:57:04.912580 23695 raft_consensus.cc:1261] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Refusing update from remote peer 93533afcf59147849cd34cac198236b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:04.912581 23891 raft_consensus.cc:1261] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Refusing update from remote peer 93533afcf59147849cd34cac198236b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:04.912957 24005 consensus_queue.cc:1035] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:04.914953 24008 mvcc.cc:205] Tried to move back new op lower bound from 7316161433239134208 to 7316161433096572928. Current Snapshot: MvccSnapshot[applied={T|T < 7316161433239134208}]
I20260808 06:57:04.915577 24011 mvcc.cc:205] Tried to move back new op lower bound from 7316161433239134208 to 7316161433096572928. Current Snapshot: MvccSnapshot[applied={T|T < 7316161433239134208}]
I20260808 06:57:04.915607 23993 consensus_queue.cc:1035] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:04.918910 23265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:04.919020 23265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:04.919795 23265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8: Sending DeleteTablet for 3 replicas of tablet 0dd65b51862743b7b5955ebacdd521b1
I20260808 06:57:04.920217 23464 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd65b51862743b7b5955ebacdd521b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:04 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:47890
I20260808 06:57:04.920248 23858 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd65b51862743b7b5955ebacdd521b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:04 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:41832
I20260808 06:57:04.920287 23668 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd65b51862743b7b5955ebacdd521b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:04 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:45950
I20260808 06:57:04.920657 24022 tablet_replica.cc:330] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: stopping tablet replica
I20260808 06:57:04.920768 24022 raft_consensus.cc:2229] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:04.920842 23265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 06:57:04.920940 24022 raft_consensus.cc:2258] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 06:57:04.920984 23265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:04.921216 24024 tablet_replica.cc:330] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: stopping tablet replica
I20260808 06:57:04.921301 24025 tablet_replica.cc:330] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: stopping tablet replica
I20260808 06:57:04.921417 24025 raft_consensus.cc:2229] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:04.921413 24024 raft_consensus.cc:2229] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:04.921542 24025 raft_consensus.cc:2258] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:04.921768 24024 raft_consensus.cc:2258] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:04.922104 24025 ts_tablet_manager.cc:1905] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:04.922605 23464 tablet_service.cc:1467] Processing CreateTablet for tablet 86ad76fc7c154eb28d7186b177840ad1 (DEFAULT_TABLE table=test-table [id=9a61aa4e334a42d3978309708d384169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:04.922720 23464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ad76fc7c154eb28d7186b177840ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.922892 23858 tablet_service.cc:1467] Processing CreateTablet for tablet 86ad76fc7c154eb28d7186b177840ad1 (DEFAULT_TABLE table=test-table [id=9a61aa4e334a42d3978309708d384169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:04.923008 23858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ad76fc7c154eb28d7186b177840ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.923744 23668 tablet_service.cc:1467] Processing CreateTablet for tablet 86ad76fc7c154eb28d7186b177840ad1 (DEFAULT_TABLE table=test-table [id=9a61aa4e334a42d3978309708d384169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:04.923853 23668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ad76fc7c154eb28d7186b177840ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.924387 24025 ts_tablet_manager.cc:1918] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:04.924446 24025 log.cc:1196] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/0dd65b51862743b7b5955ebacdd521b1
I20260808 06:57:04.924701 24025 ts_tablet_manager.cc:1939] T 0dd65b51862743b7b5955ebacdd521b1 P 93533afcf59147849cd34cac198236b9: Deleting consensus metadata
I20260808 06:57:04.924901 23990 tablet_bootstrap.cc:492] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: Bootstrap starting.
I20260808 06:57:04.925057 23245 catalog_manager.cc:4841] TS 93533afcf59147849cd34cac198236b9 (127.21.28.65:41305): tablet 0dd65b51862743b7b5955ebacdd521b1 (table test-table [id=9d502701342c4b84b53ec319b02d2e26]) successfully deleted
I20260808 06:57:04.925514 23996 tablet_bootstrap.cc:492] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: Bootstrap starting.
I20260808 06:57:04.925765 23990 tablet_bootstrap.cc:654] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.926301 23996 tablet_bootstrap.cc:654] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.926630 23988 tablet_bootstrap.cc:492] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: Bootstrap starting.
I20260808 06:57:04.927100 24022 ts_tablet_manager.cc:1905] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:04.927156 24024 ts_tablet_manager.cc:1905] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:04.927660 23988 tablet_bootstrap.cc:654] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.928786 23988 tablet_bootstrap.cc:492] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: No bootstrap required, opened a new log
I20260808 06:57:04.928778 24022 ts_tablet_manager.cc:1918] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:04.928841 23988 ts_tablet_manager.cc:1397] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 06:57:04.928880 24022 log.cc:1196] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/0dd65b51862743b7b5955ebacdd521b1
I20260808 06:57:04.929160 24022 ts_tablet_manager.cc:1939] T 0dd65b51862743b7b5955ebacdd521b1 P 2bc04e5fb05348c6ada90953e4180a0a: Deleting consensus metadata
I20260808 06:57:04.929131 23988 raft_consensus.cc:348] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.929203 23988 raft_consensus.cc:374] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.929221 23988 raft_consensus.cc:729] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93533afcf59147849cd34cac198236b9, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.929266 23988 consensus_queue.cc:260] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.929353 23988 ts_tablet_manager.cc:1428] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:04.929561 23247 catalog_manager.cc:4841] TS 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229): tablet 0dd65b51862743b7b5955ebacdd521b1 (table test-table [id=9d502701342c4b84b53ec319b02d2e26]) successfully deleted
I20260808 06:57:04.929658 24024 ts_tablet_manager.cc:1918] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:04.929731 24024 log.cc:1196] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/0dd65b51862743b7b5955ebacdd521b1
I20260808 06:57:04.930003 24024 ts_tablet_manager.cc:1939] T 0dd65b51862743b7b5955ebacdd521b1 P ff77c9e5e7ac4dedaae105557e0470c9: Deleting consensus metadata
I20260808 06:57:04.930011 23990 tablet_bootstrap.cc:492] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: No bootstrap required, opened a new log
I20260808 06:57:04.930053 23990 ts_tablet_manager.cc:1397] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 06:57:04.930138 23996 tablet_bootstrap.cc:492] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: No bootstrap required, opened a new log
I20260808 06:57:04.930190 23996 ts_tablet_manager.cc:1397] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260808 06:57:04.930227 23990 raft_consensus.cc:348] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.930300 23990 raft_consensus.cc:374] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.930323 23990 raft_consensus.cc:729] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc04e5fb05348c6ada90953e4180a0a, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.930362 23245 catalog_manager.cc:4841] TS ff77c9e5e7ac4dedaae105557e0470c9 (127.21.28.66:46425): tablet 0dd65b51862743b7b5955ebacdd521b1 (table test-table [id=9d502701342c4b84b53ec319b02d2e26]) successfully deleted
I20260808 06:57:04.930366 23990 consensus_queue.cc:260] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.930451 23990 ts_tablet_manager.cc:1428] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:04.930527 23996 raft_consensus.cc:348] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.930601 23996 raft_consensus.cc:374] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.930620 23996 raft_consensus.cc:729] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff77c9e5e7ac4dedaae105557e0470c9, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.930661 23996 consensus_queue.cc:260] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.930744 23996 ts_tablet_manager.cc:1428] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260808 06:57:04.937986 23958 tablet.cc:2369] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:04.945987 23999 raft_consensus.cc:482] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:04.946048 23999 raft_consensus.cc:504] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.946324 23999 leader_election.cc:288] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93533afcf59147849cd34cac198236b9 (127.21.28.65:41305), 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229)
I20260808 06:57:04.949795 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.946509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:41863 (local address 127.21.28.65:41305)
0808 06:57:04.946572 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.946574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.946582 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:04.946607 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.946608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.946617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.946666 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:04.946903 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.947448 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.948799 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.949372 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.949457 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.949467 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.949694 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.949737 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 06:57:04.949843 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.946461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41305 (local address 127.21.28.66:41863)
0808 06:57:04.946512 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.946522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.946561 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.946711 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.946712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.946713 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:04.946829 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.946832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.947521 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.947522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.948688 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.948690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.949453 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.949454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.949580 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.949584 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.949632 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.949681 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 06:57:04.949998 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.946649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:40105 (local address 127.21.28.67:39229)
0808 06:57:04.946690 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.946692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.946717 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:57:04.946740 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.946741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.946761 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.946807 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:04.947009 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.947814 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.949098 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.949568 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.949642 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.949650 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.949883 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.949933 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:57:04.950248 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.946485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:39229 (local address 127.21.28.66:40105)
0808 06:57:04.946635 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.946643 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.946677 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.946853 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.946854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.946857 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:04.946930 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.946932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.947907 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.947908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.949022 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.949025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.949663 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.949664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.949743 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.949757 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.950126 (+ 369us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.950172 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260808 06:57:04.950331 23492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ad76fc7c154eb28d7186b177840ad1" candidate_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93533afcf59147849cd34cac198236b9" is_pre_election: true
I20260808 06:57:04.950524 23492 raft_consensus.cc:2454] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff77c9e5e7ac4dedaae105557e0470c9 in term 0.
I20260808 06:57:04.950511 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ad76fc7c154eb28d7186b177840ad1" candidate_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc04e5fb05348c6ada90953e4180a0a" is_pre_election: true
I20260808 06:57:04.950583 23891 raft_consensus.cc:2454] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff77c9e5e7ac4dedaae105557e0470c9 in term 0.
I20260808 06:57:04.950819 23588 leader_election.cc:302] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93533afcf59147849cd34cac198236b9, ff77c9e5e7ac4dedaae105557e0470c9; no voters:
I20260808 06:57:04.950935 23999 raft_consensus.cc:2781] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:04.950970 23999 raft_consensus.cc:482] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:04.950989 23999 raft_consensus.cc:3037] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:04.952056 23999 raft_consensus.cc:504] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.952175 23999 leader_election.cc:288] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [CANDIDATE]: Term 1 election: Requested vote from peers 93533afcf59147849cd34cac198236b9 (127.21.28.65:41305), 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229)
I20260808 06:57:04.952313 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ad76fc7c154eb28d7186b177840ad1" candidate_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc04e5fb05348c6ada90953e4180a0a"
I20260808 06:57:04.952382 23891 raft_consensus.cc:3037] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:04.952766 23492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ad76fc7c154eb28d7186b177840ad1" candidate_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93533afcf59147849cd34cac198236b9"
I20260808 06:57:04.952848 23492 raft_consensus.cc:3037] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:04.953248 23891 raft_consensus.cc:2454] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff77c9e5e7ac4dedaae105557e0470c9 in term 1.
I20260808 06:57:04.953413 23589 leader_election.cc:302] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc04e5fb05348c6ada90953e4180a0a, ff77c9e5e7ac4dedaae105557e0470c9; no voters:
I20260808 06:57:04.953505 23999 raft_consensus.cc:2781] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:04.953637 23999 raft_consensus.cc:686] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 LEADER]: Becoming Leader. State: Replica: ff77c9e5e7ac4dedaae105557e0470c9, State: Running, Role: LEADER
I20260808 06:57:04.953717 23999 consensus_queue.cc:237] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.954129 23492 raft_consensus.cc:2454] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff77c9e5e7ac4dedaae105557e0470c9 in term 1.
I20260808 06:57:04.954347 23265 catalog_manager.cc:5495] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff77c9e5e7ac4dedaae105557e0470c9 (127.21.28.66). New cstate: current_term: 1 leader_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } health_report { overall_health: UNKNOWN } } }
Aug 08 06:57:04 dist-test-slave-nj21 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:04.983126 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.976417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.21.28.126:36905)
0808 06:57:04.976463 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:04.976465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:04.976611 (+ 146us) server_negotiation.cc:407] Connection header received
0808 06:57:04.976664 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:04.976665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:04.976674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:04.976719 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:04.977210 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.977680 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.978318 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.978500 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.978550 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.980902 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:04.980903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:04.980903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:04.980910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:04.982090 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.982504 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.982504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.982505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.982547 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:04.982714 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:04.982715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:04.982715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:04.982803 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:04.982847 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:04.982995 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 06:57:04.983066 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":26}
I20260808 06:57:04.983237 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:04.976375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36905 (local address 127.0.0.1:49352)
0808 06:57:04.976548 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:04.976557 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:04.976614 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:04.976728 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:04.976728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:04.977093 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:04.977135 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.977138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.977744 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.977745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.978255 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:04.978256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:04.978553 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:04.978554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:04.978602 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:04.979127 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:04.979131 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:04.980811 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:04.982165 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.982173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.982175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.982437 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.982597 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:04.982598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:04.982599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:04.982665 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:04.982851 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:04.982857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:04.982909 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 06:57:04.982970 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 06:57:04.983135 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":50,"thread_start_us":81,"threads_started":1}
I20260808 06:57:04.985280 23265 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:49352:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 06:57:04.985558 23265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:04.986809 23464 tablet_service.cc:1467] Processing CreateTablet for tablet 284efd1614584ec19493cf31e192a26e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=785eb1368d204e0eb1795246bf7a8850]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:04.986837 23858 tablet_service.cc:1467] Processing CreateTablet for tablet 284efd1614584ec19493cf31e192a26e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=785eb1368d204e0eb1795246bf7a8850]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:04.986923 23668 tablet_service.cc:1467] Processing CreateTablet for tablet 284efd1614584ec19493cf31e192a26e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=785eb1368d204e0eb1795246bf7a8850]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:04.987015 23858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284efd1614584ec19493cf31e192a26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.987015 23464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284efd1614584ec19493cf31e192a26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.987071 23668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284efd1614584ec19493cf31e192a26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:04.988761 23996 tablet_bootstrap.cc:492] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: Bootstrap starting.
I20260808 06:57:04.988768 23988 tablet_bootstrap.cc:492] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: Bootstrap starting.
I20260808 06:57:04.989543 23988 tablet_bootstrap.cc:654] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.989630 23996 tablet_bootstrap.cc:654] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.990253 23990 tablet_bootstrap.cc:492] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: Bootstrap starting.
I20260808 06:57:04.990932 23996 tablet_bootstrap.cc:492] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: No bootstrap required, opened a new log
I20260808 06:57:04.990993 23996 ts_tablet_manager.cc:1397] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 06:57:04.991183 23990 tablet_bootstrap.cc:654] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:04.991312 23996 raft_consensus.cc:348] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.991394 23996 raft_consensus.cc:374] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.991415 23996 raft_consensus.cc:729] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff77c9e5e7ac4dedaae105557e0470c9, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.991451 23996 consensus_queue.cc:260] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.991533 23996 ts_tablet_manager.cc:1428] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:04.991865 23999 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:04.991950 23999 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:04.992353 23990 tablet_bootstrap.cc:492] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: No bootstrap required, opened a new log
I20260808 06:57:04.992409 23990 ts_tablet_manager.cc:1397] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:04.992611 23988 tablet_bootstrap.cc:492] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: No bootstrap required, opened a new log
I20260808 06:57:04.992668 23988 ts_tablet_manager.cc:1397] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 06:57:04.992695 23990 raft_consensus.cc:348] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.992766 23990 raft_consensus.cc:374] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.992795 23990 raft_consensus.cc:729] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc04e5fb05348c6ada90953e4180a0a, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.992833 23990 consensus_queue.cc:260] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.992921 23990 ts_tablet_manager.cc:1428] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:04.992969 23988 raft_consensus.cc:348] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.993045 23988 raft_consensus.cc:374] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:04.993065 23988 raft_consensus.cc:729] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93533afcf59147849cd34cac198236b9, State: Initialized, Role: FOLLOWER
I20260808 06:57:04.993110 23988 consensus_queue.cc:260] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:04.993193 23988 ts_tablet_manager.cc:1428] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:04.993144 24000 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:04.993278 24000 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:04.993296 23993 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:04.993371 23993 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:05.108898 23993 raft_consensus.cc:482] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:05.108996 23993 raft_consensus.cc:504] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:05.109135 23993 leader_election.cc:288] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff77c9e5e7ac4dedaae105557e0470c9 (127.21.28.66:46425), 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229)
I20260808 06:57:05.109362 23694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284efd1614584ec19493cf31e192a26e" candidate_uuid: "93533afcf59147849cd34cac198236b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" is_pre_election: true
I20260808 06:57:05.109373 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284efd1614584ec19493cf31e192a26e" candidate_uuid: "93533afcf59147849cd34cac198236b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc04e5fb05348c6ada90953e4180a0a" is_pre_election: true
I20260808 06:57:05.109493 23891 raft_consensus.cc:2454] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93533afcf59147849cd34cac198236b9 in term 0.
I20260808 06:57:05.109493 23694 raft_consensus.cc:2454] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93533afcf59147849cd34cac198236b9 in term 0.
I20260808 06:57:05.109633 23396 leader_election.cc:302] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc04e5fb05348c6ada90953e4180a0a, 93533afcf59147849cd34cac198236b9; no voters:
I20260808 06:57:05.109754 23993 raft_consensus.cc:2781] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:05.109800 23993 raft_consensus.cc:482] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:05.109813 23993 raft_consensus.cc:3037] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:05.110584 23993 raft_consensus.cc:504] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:05.110693 23993 leader_election.cc:288] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [CANDIDATE]: Term 1 election: Requested vote from peers ff77c9e5e7ac4dedaae105557e0470c9 (127.21.28.66:46425), 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229)
I20260808 06:57:05.110874 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284efd1614584ec19493cf31e192a26e" candidate_uuid: "93533afcf59147849cd34cac198236b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc04e5fb05348c6ada90953e4180a0a"
I20260808 06:57:05.110903 23694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284efd1614584ec19493cf31e192a26e" candidate_uuid: "93533afcf59147849cd34cac198236b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff77c9e5e7ac4dedaae105557e0470c9"
I20260808 06:57:05.110961 23891 raft_consensus.cc:3037] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:05.110980 23694 raft_consensus.cc:3037] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:05.111639 23891 raft_consensus.cc:2454] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93533afcf59147849cd34cac198236b9 in term 1.
I20260808 06:57:05.111804 23694 raft_consensus.cc:2454] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93533afcf59147849cd34cac198236b9 in term 1.
I20260808 06:57:05.111873 23396 leader_election.cc:302] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc04e5fb05348c6ada90953e4180a0a, 93533afcf59147849cd34cac198236b9; no voters:
I20260808 06:57:05.111965 23993 raft_consensus.cc:2781] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:05.112008 23993 raft_consensus.cc:686] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [term 1 LEADER]: Becoming Leader. State: Replica: 93533afcf59147849cd34cac198236b9, State: Running, Role: LEADER
I20260808 06:57:05.112080 23993 consensus_queue.cc:237] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } }
I20260808 06:57:05.112411 24005 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: TxnStatusTablet state changed. Reason: New leader 93533afcf59147849cd34cac198236b9. Latest consensus state: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:05.112484 24005 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:05.112715 24054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 06:57:05.112704 23265 catalog_manager.cc:5495] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93533afcf59147849cd34cac198236b9 (127.21.28.65). New cstate: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } health_report { overall_health: UNKNOWN } } }
I20260808 06:57:05.112833 24054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 06:57:05.138803 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.136210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41305 (local address 127.0.0.1:47900)
0808 06:57:05.136298 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:05.136310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:05.136361 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:05.136527 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:05.136528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:05.136529 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:05.136578 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:05.136580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.137448 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.137449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:05.138010 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:05.138012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.138453 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.138454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:05.138503 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.138518 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:05.138669 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:05.138673 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:05.138702 (+ 29us) client_negotiation.cc:240] Negotiation successful
0808 06:57:05.138739 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 06:57:05.138819 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.136263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.21.28.65:41305)
0808 06:57:05.136333 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:05.136335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:05.136346 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:57:05.136446 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:05.136448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:05.136459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:05.136515 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:05.136634 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.137382 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.138079 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.138384 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.138441 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.138584 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:05.138620 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:05.138667 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:05.138711 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 06:57:05.138738 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:57:05.139874 23694 raft_consensus.cc:1261] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Refusing update from remote peer 93533afcf59147849cd34cac198236b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:05.139874 23891 raft_consensus.cc:1261] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Refusing update from remote peer 93533afcf59147849cd34cac198236b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:05.140084 23993 consensus_queue.cc:1035] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:05.140228 24005 consensus_queue.cc:1035] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:05.140709 24000 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: TxnStatusTablet state changed. Reason: New leader 93533afcf59147849cd34cac198236b9. Latest consensus state: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:05.140846 24000 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:05.140836 23999 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: TxnStatusTablet state changed. Reason: New leader 93533afcf59147849cd34cac198236b9. Latest consensus state: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:05.140926 23999 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:05.141382 24005 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:05.141460 24005 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:05.142246 24000 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:05.142335 24000 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P 2bc04e5fb05348c6ada90953e4180a0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:05.142292 23993 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:05.142382 23993 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P 93533afcf59147849cd34cac198236b9: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:05.142659 24017 mvcc.cc:205] Tried to move back new op lower bound from 7316161434171187200 to 7316161434059374592. Current Snapshot: MvccSnapshot[applied={T|T < 7316161434171187200}]
I20260808 06:57:05.143002 23999 tablet_replica.cc:439] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93533afcf59147849cd34cac198236b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 } } peers { permanent_uuid: "ff77c9e5e7ac4dedaae105557e0470c9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46425 } } peers { permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 } } }
I20260808 06:57:05.143081 23999 tablet_replica.cc:442] T 284efd1614584ec19493cf31e192a26e P ff77c9e5e7ac4dedaae105557e0470c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:05.146564 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.144012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.21.28.66:46425)
0808 06:57:05.144062 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:05.144063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:05.144070 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:05.144108 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:05.144108 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:05.144123 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:05.144177 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:05.144322 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.144807 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.145830 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.146017 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.146065 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.146198 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:05.146293 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:05.146338 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:05.146390 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 06:57:05.146405 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 06:57:05.152184 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.149051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:44581 (local address 127.21.28.65:41305)
0808 06:57:05.149099 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:05.149101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:05.149314 (+ 213us) server_negotiation.cc:407] Connection header received
0808 06:57:05.149348 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:05.149349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:05.149360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:05.149421 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:05.150020 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.150749 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.151417 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.151740 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.151805 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.151975 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:05.152007 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:05.152062 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:05.152096 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 06:57:05.152116 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:05.152256 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.148990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41305 (local address 127.21.28.66:44581)
0808 06:57:05.149249 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:05.149258 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:05.149307 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:05.149422 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:05.149423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:05.149424 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:05.149961 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:05.149964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.150811 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.150812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:05.151345 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:05.151347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.151801 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.151802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:05.151896 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.151909 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:05.152063 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:05.152064 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:05.152093 (+ 29us) client_negotiation.cc:240] Negotiation successful
0808 06:57:05.152128 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":128,"threads_started":1}
I20260808 06:57:05.154479 23891 raft_consensus.cc:1261] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Refusing update from remote peer ff77c9e5e7ac4dedaae105557e0470c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:05.154547 23492 raft_consensus.cc:1261] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Refusing update from remote peer ff77c9e5e7ac4dedaae105557e0470c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:05.155082 23999 consensus_queue.cc:1035] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93533afcf59147849cd34cac198236b9" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:05.155306 24033 consensus_queue.cc:1035] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc04e5fb05348c6ada90953e4180a0a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:05.174516 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.168285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:60623 (local address 127.21.28.66:46425)
0808 06:57:05.168353 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:05.168363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:05.171944 (+ 3581us) server_negotiation.cc:407] Connection header received
0808 06:57:05.171972 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:05.171972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:05.171983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:05.172055 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:05.172144 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.172884 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.173758 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.174049 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.174107 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.174247 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:05.174288 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:05.174331 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:05.174408 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 06:57:05.174430 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 06:57:05.174522 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.168087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46425 (local address 127.21.28.65:60623)
0808 06:57:05.171875 (+ 3788us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:05.171885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:05.171939 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:05.172058 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:05.172059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:05.172060 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:05.172109 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:05.172110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.172945 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.172946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:05.173652 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:05.173655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.174105 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.174106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:05.174159 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.174174 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:05.174340 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:05.174342 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:05.174402 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 06:57:05.174446 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3730,"thread_start_us":182,"threads_started":1}
I20260808 06:57:05.182122 23262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:05.182243 23262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:05.182900 23262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 706c7277b7c6404aa595eccbf67f9ec8: Sending DeleteTablet for 3 replicas of tablet 86ad76fc7c154eb28d7186b177840ad1
I20260808 06:57:05.183159 23858 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ad76fc7c154eb28d7186b177840ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:05 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:41832
I20260808 06:57:05.183314 24022 tablet_replica.cc:330] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: stopping tablet replica
I20260808 06:57:05.183360 24022 raft_consensus.cc:2229] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:05.183395 24022 raft_consensus.cc:2258] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:05.183454 23668 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ad76fc7c154eb28d7186b177840ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:05 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:45950
I20260808 06:57:05.183557 24024 tablet_replica.cc:330] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: stopping tablet replica
I20260808 06:57:05.183622 24024 raft_consensus.cc:2229] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:05.183619 23464 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ad76fc7c154eb28d7186b177840ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:05 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:47890
I20260808 06:57:05.183722 24024 raft_consensus.cc:2258] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:05.183917 24025 tablet_replica.cc:330] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: stopping tablet replica
I20260808 06:57:05.183986 24025 raft_consensus.cc:2229] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:05.184018 24025 raft_consensus.cc:2258] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:05.185038 24025 ts_tablet_manager.cc:1905] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:05.186039 24024 ts_tablet_manager.cc:1905] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:05.186913 24025 ts_tablet_manager.cc:1918] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:05.186969 24025 log.cc:1196] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/86ad76fc7c154eb28d7186b177840ad1
I20260808 06:57:05.187201 24025 ts_tablet_manager.cc:1939] T 86ad76fc7c154eb28d7186b177840ad1 P 93533afcf59147849cd34cac198236b9: Deleting consensus metadata
I20260808 06:57:05.187520 23245 catalog_manager.cc:4841] TS 93533afcf59147849cd34cac198236b9 (127.21.28.65:41305): tablet 86ad76fc7c154eb28d7186b177840ad1 (table test-table [id=9a61aa4e334a42d3978309708d384169]) successfully deleted
I20260808 06:57:05.187840 24024 ts_tablet_manager.cc:1918] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:05.187968 24022 ts_tablet_manager.cc:1905] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:05.187994 24024 log.cc:1196] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/86ad76fc7c154eb28d7186b177840ad1
I20260808 06:57:05.188267 24024 ts_tablet_manager.cc:1939] T 86ad76fc7c154eb28d7186b177840ad1 P ff77c9e5e7ac4dedaae105557e0470c9: Deleting consensus metadata
I20260808 06:57:05.189772 24022 ts_tablet_manager.cc:1918] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:05.189824 24022 log.cc:1196] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/86ad76fc7c154eb28d7186b177840ad1
I20260808 06:57:05.190007 24022 ts_tablet_manager.cc:1939] T 86ad76fc7c154eb28d7186b177840ad1 P 2bc04e5fb05348c6ada90953e4180a0a: Deleting consensus metadata
I20260808 06:57:05.190330 23245 catalog_manager.cc:4841] TS ff77c9e5e7ac4dedaae105557e0470c9 (127.21.28.66:46425): tablet 86ad76fc7c154eb28d7186b177840ad1 (table test-table [id=9a61aa4e334a42d3978309708d384169]) successfully deleted
I20260808 06:57:05.190400 23247 catalog_manager.cc:4841] TS 2bc04e5fb05348c6ada90953e4180a0a (127.21.28.67:39229): tablet 86ad76fc7c154eb28d7186b177840ad1 (table test-table [id=9a61aa4e334a42d3978309708d384169]) successfully deleted
Aug 08 06:57:05 dist-test-slave-nj21 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:05.202538 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.185055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.21.28.65:41305)
0808 06:57:05.186750 (+ 1695us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:05.186752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:05.191872 (+ 5120us) server_negotiation.cc:407] Connection header received
0808 06:57:05.191963 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:05.191964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:05.191975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:05.192002 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:05.192789 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.193478 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.194670 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.194943 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.194985 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.199082 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:05.199089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:05.199089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:05.199097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:05.201175 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:05.201634 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:05.201638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:05.201638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:05.201712 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:05.201939 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:05.201940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:05.201940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:05.202119 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:05.202272 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:05.202364 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 06:57:05.202447 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1672}
W20260808 06:57:05.202836 23546 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47906
I20260808 06:57:05.211514 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:05.203914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.21.28.126:36905)
0808 06:57:05.203968 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:05.203970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:05.204086 (+ 116us) server_negotiation.cc:407] Connection header received
0808 06:57:05.204177 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:05.204178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:05.204190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:05.204217 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:05.204874 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.205611 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.206588 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:05.206877 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:05.206921 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:05.208455 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:05.208457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:05.208457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:05.208464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:05.210122 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:05.210559 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:05.210561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:05.210561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:05.210666 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:05.210884 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:05.210885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:05.210885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:05.211012 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:05.211082 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:05.211328 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 06:57:05.211419 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260808 06:57:05.212188 23262 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49356
I20260808 06:57:05.212865 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23371
I20260808 06:57:05.228492 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23563
I20260808 06:57:05.235600 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23768
I20260808 06:57:05.244390 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23222
2026-08-08T06:57:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2430 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:05 dist-test-slave-nj21 krb5kdc[24083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:05 dist-test-slave-nj21 krb5kdc[24083](info): set up 2 sockets
Aug 08 06:57:05 dist-test-slave-nj21 krb5kdc[24083](info): commencing operation
krb5kdc: starting...
W20260808 06:57:06.372285 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:06 dist-test-slave-nj21 krb5kdc[24083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:06.481678 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:33957
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:33957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:06.561414 24099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:06.561599 24099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:06.561618 24099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:06.562669 24099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:06.562700 24099 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 06:57:06.562712 24099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:06.562723 24099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:06.562734 24099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:06.562747 24099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:06.564190 24099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:33957
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:33957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:06.564381 24099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:06.564548 24099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:06.569023 24104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:06.569031 24108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:06.569140 24099 server_base.cc:1034] running on GCE node
W20260808 06:57:06.569022 24105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:06.569423 24099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:06.569720 24099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:06.570858 24099 hybrid_clock.cc:648] HybridClock initialized: now 1786172226570841 us; error 31 us; skew 500 ppm
Aug 08 06:57:06 dist-test-slave-nj21 krb5kdc[24083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:06.573179 24099 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:06.573583 24099 webserver.cc:460] Webserver started at http://127.21.28.126:33819/ using document root <none> and password file <none>
I20260808 06:57:06.573793 24099 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:06.573840 24099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:06.574003 24099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:06.574980 24099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "8a24177cb2954ee18e2f8bfc35ee506e"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "52649fcf9825ed0accc4967e208bb5f9"
server_key_iv: "55c047af0e2f1ec7d10c67929cd23dfe"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.575289 24099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "8a24177cb2954ee18e2f8bfc35ee506e"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "52649fcf9825ed0accc4967e208bb5f9"
server_key_iv: "55c047af0e2f1ec7d10c67929cd23dfe"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.576529 24099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:06.577157 24118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:06.577301 24099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:06.577384 24099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "8a24177cb2954ee18e2f8bfc35ee506e"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "52649fcf9825ed0accc4967e208bb5f9"
server_key_iv: "55c047af0e2f1ec7d10c67929cd23dfe"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.577445 24099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:06.586468 24099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:06.588455 24099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:06.588581 24099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:06.592939 24099 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:33957
I20260808 06:57:06.592978 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:33957 every 8 connection(s)
I20260808 06:57:06.593247 24099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:06.593814 24206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:06.595692 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 24099
I20260808 06:57:06.595783 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:06.595966 21617 external_mini_cluster.cc:1428] Setting key 784eb5e5b20fc720e6eebc540aa19fd3
I20260808 06:57:06.596084 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: Bootstrap starting.
I20260808 06:57:06.596719 24206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:06.596979 24206 log.cc:824] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:06.597712 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: No bootstrap required, opened a new log
I20260808 06:57:06.599835 24206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24177cb2954ee18e2f8bfc35ee506e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33957 } }
I20260808 06:57:06.600000 24206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:06.600026 24206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a24177cb2954ee18e2f8bfc35ee506e, State: Initialized, Role: FOLLOWER
I20260808 06:57:06.600142 24206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24177cb2954ee18e2f8bfc35ee506e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33957 } }
I20260808 06:57:06.600207 24206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:06.600229 24206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:06.600273 24206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:06.600890 24206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24177cb2954ee18e2f8bfc35ee506e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33957 } }
I20260808 06:57:06.600996 24206 leader_election.cc:302] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a24177cb2954ee18e2f8bfc35ee506e; no voters:
I20260808 06:57:06.601152 24206 leader_election.cc:288] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:06.601183 24212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:06.601336 24212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [term 1 LEADER]: Becoming Leader. State: Replica: 8a24177cb2954ee18e2f8bfc35ee506e, State: Running, Role: LEADER
I20260808 06:57:06.601401 24206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:06.601440 24212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24177cb2954ee18e2f8bfc35ee506e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33957 } }
I20260808 06:57:06.601732 24212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a24177cb2954ee18e2f8bfc35ee506e. Latest consensus state: current_term: 1 leader_uuid: "8a24177cb2954ee18e2f8bfc35ee506e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24177cb2954ee18e2f8bfc35ee506e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33957 } } }
I20260808 06:57:06.601815 24212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:06.601742 24215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a24177cb2954ee18e2f8bfc35ee506e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a24177cb2954ee18e2f8bfc35ee506e" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33957 } } }
I20260808 06:57:06.601912 24215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e [sys.catalog]: This master's current role is: LEADER
Aug 08 06:57:06 dist-test-slave-nj21 krb5kdc[24083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
W20260808 06:57:06.603196 24238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:57:06.603251 24238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:57:06.603293 24240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:06.603463 24240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:06.605013 24240 catalog_manager.cc:1347] Generated new cluster ID: 539cefdacc5e4f2fa339ad36ea3e05ff
I20260808 06:57:06.605064 24240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:06.606413 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.596421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.21.28.126:33957)
0808 06:57:06.596577 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:06.596586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:06.596613 (+ 27us) server_negotiation.cc:407] Connection header received
0808 06:57:06.597048 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:06.597055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:06.597110 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:06.597248 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:06.597930 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.598798 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.599867 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.600352 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.600447 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.602935 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:06.602939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:06.602939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:06.602949 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:06.604628 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.605020 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.605022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.605023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.605120 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.605313 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.605313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.605314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.605583 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:06.605701 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:06.606089 (+ 388us) server_negotiation.cc:299] Negotiation successful
0808 06:57:06.606209 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260808 06:57:06.627900 24240 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:06.628593 24240 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:06.635167 24240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: Generated new TSK 0
I20260808 06:57:06.635326 24240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:06.662019 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33957
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:06.740645 24245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:06.740832 24245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:06.740855 24245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:06.741966 24245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:06.742007 24245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:06.742026 24245 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 06:57:06.742048 24245 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:06.742071 24245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:06.743499 24245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:06.743746 24245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:06.743985 24245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:06.748675 24251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:06.748795 24245 server_base.cc:1034] running on GCE node
W20260808 06:57:06.748710 24253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:06.748685 24250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:06.749074 24245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:06.749265 24245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:06.750403 24245 hybrid_clock.cc:648] HybridClock initialized: now 1786172226750378 us; error 32 us; skew 500 ppm
Aug 08 06:57:06 dist-test-slave-nj21 krb5kdc[24083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:06.752765 24245 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:06.753141 24245 webserver.cc:460] Webserver started at http://127.21.28.65:33011/ using document root <none> and password file <none>
I20260808 06:57:06.753346 24245 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:06.753403 24245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:06.753527 24245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:06.754500 24245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "49207139453e4953a409a65efe167400"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "4dafe22f1c897521f8842e111893213f"
server_key_iv: "1661638fcd1b459b639dcf8b1ade2773"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.754815 24245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "49207139453e4953a409a65efe167400"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "4dafe22f1c897521f8842e111893213f"
server_key_iv: "1661638fcd1b459b639dcf8b1ade2773"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.756072 24245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:06.756721 24265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:06.756862 24245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:06.756944 24245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "49207139453e4953a409a65efe167400"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "4dafe22f1c897521f8842e111893213f"
server_key_iv: "1661638fcd1b459b639dcf8b1ade2773"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.757002 24245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:06.780722 24245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:06.782330 24245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:06.782465 24245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:06.782914 24245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:06.782954 24245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:06.782986 24245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:06.783010 24245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:06 dist-test-slave-nj21 krb5kdc[24083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:06.789770 24245 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:37195
I20260808 06:57:06.789851 24417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:37195 every 8 connection(s)
I20260808 06:57:06.790108 24245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:06.792202 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.783371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:53999 (local address 127.21.28.126:33957)
0808 06:57:06.783420 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:06.783422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:06.783661 (+ 239us) server_negotiation.cc:407] Connection header received
0808 06:57:06.784141 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:06.784142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:06.784157 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:06.784227 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:06.784941 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.785692 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.786837 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.787145 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.787214 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.789799 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:06.789801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:06.789801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:06.789809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:06.791143 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.791476 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.791477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.791477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.791523 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.791695 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.791695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.791696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.791807 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:06.791857 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:06.792069 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 06:57:06.792137 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 06:57:06.792346 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.783374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33957 (local address 127.21.28.65:53999)
0808 06:57:06.783530 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:06.783563 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:06.784042 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:06.784299 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:06.784303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:06.784584 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:06.784837 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:06.784843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.785778 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.785779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:06.786756 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:06.786758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.787246 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.787247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:06.787406 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.787973 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:06.787982 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:06.789709 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:06.791211 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:06.791217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:06.791219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:06.791420 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:06.791560 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:06.791560 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:06.791560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:06.791639 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:06.791855 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:06.791859 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:06.791972 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 06:57:06.792052 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 06:57:06.792161 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260808 06:57:06.792832 24418 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33957
I20260808 06:57:06.792912 24418 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:06.793084 24418 heartbeater.cc:511] Master 127.21.28.126:33957 requested a full tablet report, sending...
I20260808 06:57:06.793608 24137 ts_manager.cc:194] Registered new tserver with Master: 49207139453e4953a409a65efe167400 (127.21.28.65:37195)
I20260808 06:57:06.794406 24137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:53999
I20260808 06:57:06.796535 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 24245
I20260808 06:57:06.796638 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:06.796821 21617 external_mini_cluster.cc:1428] Setting key 6785c80536a35f0bd2ae043b32b90b15
I20260808 06:57:06.798663 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.793772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33957 (local address 127.21.28.65:57619)
0808 06:57:06.793902 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:06.793913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:06.793961 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:06.794082 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:06.794083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:06.794216 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:06.794270 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:06.794271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.795018 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.795019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:06.795543 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:06.795546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.795879 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.795880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:06.795933 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.796235 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:06.796239 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:06.796479 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:06.797868 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:06.797869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:06.797870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:06.798024 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:06.798201 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:06.798202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:06.798202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:06.798249 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:06.798464 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:06.798464 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:06.798496 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 06:57:06.798537 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 06:57:06.798605 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260808 06:57:06.798700 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.793838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:57619 (local address 127.21.28.126:33957)
0808 06:57:06.793935 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:06.793937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:06.793948 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:57:06.794006 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:06.794007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:06.794015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:06.794070 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:06.794322 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.794911 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.795603 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.795823 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.795866 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.796555 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:06.796556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:06.796557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:06.796571 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:06.797791 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.798088 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.798089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.798089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.798147 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.798295 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.798296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.798296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.798414 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:06.798473 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:06.798561 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 06:57:06.798627 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:06.820091 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33957
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:06.898587 24430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:06.898770 24430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:06.898792 24430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:06.899863 24430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:06.899901 24430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:06.899920 24430 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 06:57:06.899936 24430 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:06.899951 24430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:06.901283 24430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:06.901444 24430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:06.901638 24430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:06.906297 24446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:06.906273 24438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:06.906280 24442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:06.906495 24430 server_base.cc:1034] running on GCE node
I20260808 06:57:06.906682 24430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:06.906893 24430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:06.908025 24430 hybrid_clock.cc:648] HybridClock initialized: now 1786172226908008 us; error 29 us; skew 500 ppm
Aug 08 06:57:06 dist-test-slave-nj21 krb5kdc[24083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:06.910279 24430 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:06.910660 24430 webserver.cc:460] Webserver started at http://127.21.28.66:37687/ using document root <none> and password file <none>
I20260808 06:57:06.910838 24430 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:06.910887 24430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:06.911000 24430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:06.911971 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "1a918e2e600d44268412dc6196c1f777"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "5c5e8d09bcf0a8cda06538143d4893eb"
server_key_iv: "424c76959aac53e9be978d93be507e62"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.912333 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "1a918e2e600d44268412dc6196c1f777"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "5c5e8d09bcf0a8cda06538143d4893eb"
server_key_iv: "424c76959aac53e9be978d93be507e62"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.913527 24430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:06.914152 24455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:06.914287 24430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:06.914363 24430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "1a918e2e600d44268412dc6196c1f777"
format_stamp: "Formatted at 2026-08-08 06:57:06 on dist-test-slave-nj21"
server_key: "5c5e8d09bcf0a8cda06538143d4893eb"
server_key_iv: "424c76959aac53e9be978d93be507e62"
server_key_version: "encryptionkey@0"
I20260808 06:57:06.914425 24430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:06.937893 24430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:06.939462 24430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:06.939586 24430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:06.940080 24430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:06.940122 24430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:06.940142 24430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:06.940155 24430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:06 dist-test-slave-nj21 krb5kdc[24083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:06.947283 24430 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:45525
I20260808 06:57:06.947362 24614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:45525 every 8 connection(s)
I20260808 06:57:06.947662 24430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:06.949882 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.940496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:48749 (local address 127.21.28.126:33957)
0808 06:57:06.940561 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:06.940563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:06.940801 (+ 238us) server_negotiation.cc:407] Connection header received
0808 06:57:06.941445 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:06.941446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:06.941456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:06.941512 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:06.942307 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.943081 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.944550 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.944854 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.944934 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.947309 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:06.947310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:06.947310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:06.947318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:06.948649 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.949006 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.949007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.949008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.949068 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.949278 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.949279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.949279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.949401 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:06.949466 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:06.949711 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 06:57:06.949792 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:57:06.950040 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.940477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33957 (local address 127.21.28.66:48749)
0808 06:57:06.940659 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:06.940704 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:06.941342 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:06.941575 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:06.941580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:06.941928 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:06.942210 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:06.942217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.943181 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.943182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:06.944465 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:06.944468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.944954 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.944955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:06.945101 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.945660 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:06.945670 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:06.947216 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:06.948719 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:06.948725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:06.948727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:06.948939 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:06.949120 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:06.949121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:06.949121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:06.949224 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:06.949465 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:06.949471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:06.949598 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 06:57:06.949683 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 06:57:06.949826 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260808 06:57:06.950472 24615 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33957
I20260808 06:57:06.950551 24615 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:06.950707 24615 heartbeater.cc:511] Master 127.21.28.126:33957 requested a full tablet report, sending...
I20260808 06:57:06.951076 24138 ts_manager.cc:194] Registered new tserver with Master: 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:06.951529 24138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:48749
I20260808 06:57:06.954159 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 24430
I20260808 06:57:06.954257 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:06.954435 21617 external_mini_cluster.cc:1428] Setting key 7674a72396da82e78a4f123e1762b9c1
I20260808 06:57:06.956447 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.951423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:40203 (local address 127.21.28.126:33957)
0808 06:57:06.951461 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:06.951463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:06.951576 (+ 113us) server_negotiation.cc:407] Connection header received
0808 06:57:06.951641 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:06.951642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:06.951651 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:06.951737 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:06.952065 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.952588 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.953190 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.953352 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.953399 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.954169 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:06.954170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:06.954171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:06.954176 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:06.955459 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.955828 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.955829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.955829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.955874 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:06.956040 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:06.956041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:06.956041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:06.956161 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:06.956206 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:06.956335 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 06:57:06.956391 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 06:57:06.956559 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:06.951359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33957 (local address 127.21.28.66:40203)
0808 06:57:06.951513 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:06.951523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:06.951570 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:06.951770 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:06.951771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:06.951936 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:06.951991 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:06.951992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.952658 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.952658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:06.953139 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:06.953140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:06.953402 (+ 262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:06.953403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:06.953471 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:06.953844 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:06.953848 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:06.954102 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:06.955537 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:06.955538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:06.955539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:06.955762 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:06.955927 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:06.955927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:06.955928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:06.955989 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:06.956208 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:06.956209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:06.956251 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 06:57:06.956317 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 06:57:06.956415 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:06.978397 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33957
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:07.057745 24631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:07.057945 24631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:07.057967 24631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:07.059042 24631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:07.059075 24631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:07.059094 24631 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 06:57:07.059111 24631 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:07.059127 24631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:07.060575 24631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:07.060772 24631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:07.060951 24631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:07.065606 24642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:07.065611 24638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:07.065611 24640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:07.065816 24631 server_base.cc:1034] running on GCE node
I20260808 06:57:07.066015 24631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:07.066233 24631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:07.067381 24631 hybrid_clock.cc:648] HybridClock initialized: now 1786172227067363 us; error 30 us; skew 500 ppm
Aug 08 06:57:07 dist-test-slave-nj21 krb5kdc[24083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:07.069871 24631 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:07.070277 24631 webserver.cc:460] Webserver started at http://127.21.28.67:39353/ using document root <none> and password file <none>
I20260808 06:57:07.070492 24631 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:07.070538 24631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:07.070660 24631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:07.071610 24631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "487586eb30df460e9d3416738c9f1314"
format_stamp: "Formatted at 2026-08-08 06:57:07 on dist-test-slave-nj21"
server_key: "42523e43cab9aa207497d5aaac41fc4f"
server_key_iv: "56b5efc84323d0893d7f9a9ee897064b"
server_key_version: "encryptionkey@0"
I20260808 06:57:07.071988 24631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "487586eb30df460e9d3416738c9f1314"
format_stamp: "Formatted at 2026-08-08 06:57:07 on dist-test-slave-nj21"
server_key: "42523e43cab9aa207497d5aaac41fc4f"
server_key_iv: "56b5efc84323d0893d7f9a9ee897064b"
server_key_version: "encryptionkey@0"
I20260808 06:57:07.073310 24631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:07.073921 24651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:07.074074 24631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:07.074148 24631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "487586eb30df460e9d3416738c9f1314"
format_stamp: "Formatted at 2026-08-08 06:57:07 on dist-test-slave-nj21"
server_key: "42523e43cab9aa207497d5aaac41fc4f"
server_key_iv: "56b5efc84323d0893d7f9a9ee897064b"
server_key_version: "encryptionkey@0"
I20260808 06:57:07.074213 24631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:07.090544 24631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:07.092592 24631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:07.092722 24631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:07.093276 24631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:07.093318 24631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:07.093348 24631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:07.093370 24631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:07 dist-test-slave-nj21 krb5kdc[24083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:07.100512 24631 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:42069
I20260808 06:57:07.100585 24819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:42069 every 8 connection(s)
I20260808 06:57:07.100917 24631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:07.102717 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 24631
I20260808 06:57:07.102813 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:07.103051 21617 external_mini_cluster.cc:1428] Setting key 68781469e093800a5ebdff80866bd665
I20260808 06:57:07.103132 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.093659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:49013 (local address 127.21.28.126:33957)
0808 06:57:07.093708 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.093711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.093914 (+ 203us) server_negotiation.cc:407] Connection header received
0808 06:57:07.094380 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.094382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.094397 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.094484 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:07.095069 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.095838 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.097317 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.097628 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.097699 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.100240 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:07.100242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:07.100243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:07.100250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:07.101762 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:07.102168 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:07.102169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:07.102169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:07.102220 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:07.102438 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:07.102438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:07.102439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:07.102566 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:07.102614 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.102960 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.103046 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:07.103284 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.093628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33957 (local address 127.21.28.67:49013)
0808 06:57:07.093775 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.093814 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.094286 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.094532 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.094536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.094752 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:07.094983 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.094987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.095938 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.095939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.097217 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.097222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.097734 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.097736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.097885 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.098380 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:07.098389 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:07.100148 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:07.101824 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:07.101828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:07.101830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:07.102103 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:07.102267 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:07.102268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:07.102268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:07.102371 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:07.102619 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:07.102624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:07.102797 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.102919 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.103080 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260808 06:57:07.103726 24820 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33957
I20260808 06:57:07.103804 24820 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:07.103976 24820 heartbeater.cc:511] Master 127.21.28.126:33957 requested a full tablet report, sending...
I20260808 06:57:07.104329 24137 ts_manager.cc:194] Registered new tserver with Master: 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069)
I20260808 06:57:07.104789 24137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:49013
I20260808 06:57:07.109293 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.104703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:50661 (local address 127.21.28.126:33957)
0808 06:57:07.104738 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.104739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.104854 (+ 115us) server_negotiation.cc:407] Connection header received
0808 06:57:07.104941 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.104942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.104952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.104977 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:07.105368 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.105827 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.106553 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.106717 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.106774 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.107495 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:07.107496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:07.107496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:07.107501 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:07.108559 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:07.108828 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:07.108828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:07.108828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:07.108874 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:07.108984 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:07.108984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:07.108985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:07.109064 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:07.109104 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.109185 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.109235 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 06:57:07.109301 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.104649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33957 (local address 127.21.28.67:50661)
0808 06:57:07.104792 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.104802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.104855 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.105038 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.105039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.105223 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:07.105283 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.105285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.105907 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.105908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.106496 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.106498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.106762 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.106763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.106810 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.107163 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:07.107168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:07.107425 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:07.108623 (+ 1198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:07.108624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:07.108624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:07.108775 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:07.108911 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:07.108912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:07.108912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:07.108952 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:07.109097 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:07.109097 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:07.109127 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.109168 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.109233 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260808 06:57:07.114822 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 06:57:07 dist-test-slave-nj21 krb5kdc[24083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 06:57:07 dist-test-slave-nj21 krb5kdc[24083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:07.136444 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.130474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.21.28.126:33957)
0808 06:57:07.130528 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.130530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.130539 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:07.130579 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.130580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.130588 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.130633 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:07.131115 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.131591 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.132280 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.132459 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.132504 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.134406 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:07.134408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:07.134408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:07.134414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:07.135566 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:07.135902 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:07.135902 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:07.135903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:07.135954 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:07.136084 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:07.136085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:07.136085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:07.136178 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:07.136245 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.136318 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.136378 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:57:07.137694 24137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:07.139218 24137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:07.145499 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.141378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:45525 (local address 127.0.0.1:54550)
0808 06:57:07.141644 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.141654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.141707 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.142150 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.142150 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.142152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:07.142192 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.142193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.143548 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.143549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.144502 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.144505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.145187 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.145188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.145265 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.145278 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.145341 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.145420 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":44,"thread_start_us":90,"threads_started":1}
I20260808 06:57:07.145553 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.141571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:37195 (local address 127.0.0.1:44742)
0808 06:57:07.141776 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.141784 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.141816 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.142254 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.142255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.142257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:07.142302 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.142304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.143135 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.143136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.144337 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.144340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.145301 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.145301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.145372 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.145375 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.145428 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.145462 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260808 06:57:07.145593 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.141785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.21.28.65:37195)
0808 06:57:07.142115 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.142120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.142129 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:07.142162 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.142164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.142179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.142256 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:07.142373 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.143057 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.144421 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.145193 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.145297 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.145359 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.145444 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.145489 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":97,"threads_started":1}
I20260808 06:57:07.145695 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.141372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:42069 (local address 127.0.0.1:50200)
0808 06:57:07.141548 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.141591 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.141652 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.141916 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.141919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.141921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:07.141985 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.141986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.143487 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.143488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.144593 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.144595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.145494 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.145495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.145557 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.145561 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.145590 (+ 29us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.145630 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":33,"thread_start_us":59,"threads_started":1}
I20260808 06:57:07.145985 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.141472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.21.28.67:42069)
0808 06:57:07.141777 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.141781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.141789 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:07.141817 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.141819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.141830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.141895 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:07.142060 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.143405 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.144663 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.145404 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.145788 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.145851 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.145874 (+ 23us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.145912 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":85,"threads_started":1}
I20260808 06:57:07.146270 24329 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ba11791224c16838a36f2b6b88de7 (DEFAULT_TABLE table=test-table [id=56222b0734e64bd09553466a97d9686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:07.146452 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.141499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.21.28.66:45525)
0808 06:57:07.141995 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.141999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.142006 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:07.142061 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.142063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.142077 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.142153 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:07.142388 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.143456 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.144583 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.145111 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.145176 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.145236 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.146307 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.146354 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":79,"threads_started":1}
I20260808 06:57:07.146677 24329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ba11791224c16838a36f2b6b88de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.146595 24724 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ba11791224c16838a36f2b6b88de7 (DEFAULT_TABLE table=test-table [id=56222b0734e64bd09553466a97d9686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:07.146924 24724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ba11791224c16838a36f2b6b88de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.147270 24521 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ba11791224c16838a36f2b6b88de7 (DEFAULT_TABLE table=test-table [id=56222b0734e64bd09553466a97d9686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:07.147589 24521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ba11791224c16838a36f2b6b88de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.150391 24856 tablet_bootstrap.cc:492] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Bootstrap starting.
I20260808 06:57:07.150915 24856 tablet_bootstrap.cc:654] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.151104 24858 tablet_bootstrap.cc:492] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Bootstrap starting.
I20260808 06:57:07.151258 24851 tablet_bootstrap.cc:492] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Bootstrap starting.
I20260808 06:57:07.151517 24856 log.cc:824] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:07.151868 24858 tablet_bootstrap.cc:654] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.152181 24851 tablet_bootstrap.cc:654] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.152184 24858 log.cc:824] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:07.152493 24851 log.cc:824] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:07.153220 24851 tablet_bootstrap.cc:492] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: No bootstrap required, opened a new log
I20260808 06:57:07.153287 24851 ts_tablet_manager.cc:1397] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:07.152359 24856 tablet_bootstrap.cc:492] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: No bootstrap required, opened a new log
I20260808 06:57:07.153403 24858 tablet_bootstrap.cc:492] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: No bootstrap required, opened a new log
I20260808 06:57:07.153455 24858 ts_tablet_manager.cc:1397] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:07.153461 24856 ts_tablet_manager.cc:1397] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 06:57:07.155251 24851 raft_consensus.cc:348] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.155262 24858 raft_consensus.cc:348] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.155419 24858 raft_consensus.cc:374] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.155419 24851 raft_consensus.cc:374] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.155457 24851 raft_consensus.cc:729] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487586eb30df460e9d3416738c9f1314, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.155457 24858 raft_consensus.cc:729] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a918e2e600d44268412dc6196c1f777, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.155406 24856 raft_consensus.cc:348] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.155551 24856 raft_consensus.cc:374] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.155588 24856 raft_consensus.cc:729] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49207139453e4953a409a65efe167400, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.155575 24858 consensus_queue.cc:260] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.155616 24851 consensus_queue.cc:260] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.155768 24856 consensus_queue.cc:260] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.155871 24858 ts_tablet_manager.cc:1428] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260808 06:57:07.155871 24851 ts_tablet_manager.cc:1428] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 06:57:07.155984 24820 heartbeater.cc:503] Master 127.21.28.126:33957 was elected leader, sending a full tablet report...
I20260808 06:57:07.155990 24615 heartbeater.cc:503] Master 127.21.28.126:33957 was elected leader, sending a full tablet report...
I20260808 06:57:07.156045 24856 ts_tablet_manager.cc:1428] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 06:57:07.156154 24418 heartbeater.cc:503] Master 127.21.28.126:33957 was elected leader, sending a full tablet report...
W20260808 06:57:07.198292 24616 tablet.cc:2369] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:07.262413 24870 raft_consensus.cc:482] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:07.262547 24870 raft_consensus.cc:504] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.262878 24870 leader_election.cc:288] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069), 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:07.266592 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.263116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:42155 (local address 127.21.28.67:42069)
0808 06:57:07.263179 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.263181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.263195 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:57:07.263225 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.263225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.263235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.263289 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:07.263456 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.264245 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.265684 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.266247 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.266309 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.266316 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.266494 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.266533 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:57:07.266584 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.263157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:33587 (local address 127.21.28.66:45525)
0808 06:57:07.263195 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.263197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.263283 (+ 86us) server_negotiation.cc:407] Connection header received
0808 06:57:07.263316 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.263317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.263327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.263372 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:07.263517 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.264244 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.265684 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.266244 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.266308 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.266316 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.266475 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.266516 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:07.266595 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.263049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:42069 (local address 127.21.28.65:42155)
0808 06:57:07.263112 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.263124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.263171 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.263304 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.263305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.263306 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:07.263387 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.263389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.264319 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.264320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.265602 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.265606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.266316 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.266317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.266434 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.266437 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.266485 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.266517 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":33}
I20260808 06:57:07.266603 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.263050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:45525 (local address 127.21.28.65:33587)
0808 06:57:07.263218 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.263226 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.263276 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.263381 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.263382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.263383 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:07.263455 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.263457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.264320 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.264320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.265602 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.265606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.266318 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.266318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.266410 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.266415 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.266465 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.266505 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":93,"thread_start_us":66,"threads_started":1}
I20260808 06:57:07.266976 24751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ba11791224c16838a36f2b6b88de7" candidate_uuid: "49207139453e4953a409a65efe167400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487586eb30df460e9d3416738c9f1314" is_pre_election: true
I20260808 06:57:07.267141 24751 raft_consensus.cc:2454] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49207139453e4953a409a65efe167400 in term 0.
I20260808 06:57:07.267118 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ba11791224c16838a36f2b6b88de7" candidate_uuid: "49207139453e4953a409a65efe167400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a918e2e600d44268412dc6196c1f777" is_pre_election: true
I20260808 06:57:07.267246 24551 raft_consensus.cc:2454] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49207139453e4953a409a65efe167400 in term 0.
I20260808 06:57:07.267354 24266 leader_election.cc:302] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487586eb30df460e9d3416738c9f1314, 49207139453e4953a409a65efe167400; no voters:
I20260808 06:57:07.267486 24870 raft_consensus.cc:2781] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:07.267560 24870 raft_consensus.cc:482] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:07.267576 24870 raft_consensus.cc:3037] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.268292 24870 raft_consensus.cc:504] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.268425 24870 leader_election.cc:288] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [CANDIDATE]: Term 1 election: Requested vote from peers 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069), 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:07.268569 24751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ba11791224c16838a36f2b6b88de7" candidate_uuid: "49207139453e4953a409a65efe167400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487586eb30df460e9d3416738c9f1314"
I20260808 06:57:07.268570 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ba11791224c16838a36f2b6b88de7" candidate_uuid: "49207139453e4953a409a65efe167400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a918e2e600d44268412dc6196c1f777"
I20260808 06:57:07.268651 24751 raft_consensus.cc:3037] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.268651 24551 raft_consensus.cc:3037] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.269543 24751 raft_consensus.cc:2454] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49207139453e4953a409a65efe167400 in term 1.
I20260808 06:57:07.269543 24551 raft_consensus.cc:2454] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49207139453e4953a409a65efe167400 in term 1.
I20260808 06:57:07.269703 24268 leader_election.cc:302] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a918e2e600d44268412dc6196c1f777, 49207139453e4953a409a65efe167400; no voters:
I20260808 06:57:07.269780 24870 raft_consensus.cc:2781] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:07.269930 24870 raft_consensus.cc:686] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 1 LEADER]: Becoming Leader. State: Replica: 49207139453e4953a409a65efe167400, State: Running, Role: LEADER
I20260808 06:57:07.270015 24870 consensus_queue.cc:237] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } }
I20260808 06:57:07.270829 24137 catalog_manager.cc:5495] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49207139453e4953a409a65efe167400 (127.21.28.65). New cstate: current_term: 1 leader_uuid: "49207139453e4953a409a65efe167400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } health_report { overall_health: HEALTHY } } }
W20260808 06:57:07.290743 24419 tablet.cc:2369] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 06:57:07 dist-test-slave-nj21 krb5kdc[24083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172226, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:07.314247 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.308221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33957 (local address 127.0.0.1:49398)
0808 06:57:07.308378 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.308388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.308446 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.308570 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.308571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.308863 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:07.308909 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.308911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.309529 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.309530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.310040 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.310041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.310355 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.310356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.310422 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.310846 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:07.310850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:07.312115 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:07.313308 (+ 1193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:07.313314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:07.313317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:07.313500 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:07.313660 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:07.313661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:07.313662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:07.313712 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:07.313914 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:07.313921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:07.313966 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.314023 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.314135 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260808 06:57:07.314298 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.308281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.21.28.126:33957)
0808 06:57:07.308332 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.308334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.308440 (+ 106us) server_negotiation.cc:407] Connection header received
0808 06:57:07.308499 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.308500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.308510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.308557 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:07.308980 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.309462 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.310104 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.310285 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.310335 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.312190 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:07.312191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:07.312191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:07.312197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:07.313237 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:07.313561 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:07.313562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:07.313562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:07.313611 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:07.313758 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:07.313759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:07.313759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:07.313857 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:07.313918 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.314051 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.314197 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":26}
I20260808 06:57:07.315884 24137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:49398:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 06:57:07.316075 24137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:07.317373 24329 tablet_service.cc:1467] Processing CreateTablet for tablet e6cce9c8f39045f1be85b6888d1fe06d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db4f6fecf78a4899b23db8bd54d67829]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:07.317413 24521 tablet_service.cc:1467] Processing CreateTablet for tablet e6cce9c8f39045f1be85b6888d1fe06d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db4f6fecf78a4899b23db8bd54d67829]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:07.317461 24724 tablet_service.cc:1467] Processing CreateTablet for tablet e6cce9c8f39045f1be85b6888d1fe06d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db4f6fecf78a4899b23db8bd54d67829]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:07.317551 24329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cce9c8f39045f1be85b6888d1fe06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.317557 24521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cce9c8f39045f1be85b6888d1fe06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.317595 24724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cce9c8f39045f1be85b6888d1fe06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.318785 24858 tablet_bootstrap.cc:492] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: Bootstrap starting.
I20260808 06:57:07.319044 24851 tablet_bootstrap.cc:492] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: Bootstrap starting.
I20260808 06:57:07.319525 24858 tablet_bootstrap.cc:654] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.319612 24851 tablet_bootstrap.cc:654] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.319995 24856 tablet_bootstrap.cc:492] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: Bootstrap starting.
I20260808 06:57:07.320231 24858 tablet_bootstrap.cc:492] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: No bootstrap required, opened a new log
I20260808 06:57:07.320286 24858 ts_tablet_manager.cc:1397] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 06:57:07.320465 24858 raft_consensus.cc:348] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.320549 24858 raft_consensus.cc:374] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.320602 24858 raft_consensus.cc:729] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a918e2e600d44268412dc6196c1f777, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.320627 24856 tablet_bootstrap.cc:654] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.320653 24858 consensus_queue.cc:260] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.320765 24858 ts_tablet_manager.cc:1428] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:07.321002 24865 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.321090 24865 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:07.321152 24851 tablet_bootstrap.cc:492] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: No bootstrap required, opened a new log
I20260808 06:57:07.321195 24851 ts_tablet_manager.cc:1397] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:07.321342 24851 raft_consensus.cc:348] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.321417 24851 raft_consensus.cc:374] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.321422 24856 tablet_bootstrap.cc:492] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: No bootstrap required, opened a new log
I20260808 06:57:07.321436 24851 raft_consensus.cc:729] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487586eb30df460e9d3416738c9f1314, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.321462 24856 ts_tablet_manager.cc:1397] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:07.321478 24851 consensus_queue.cc:260] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.321563 24851 ts_tablet_manager.cc:1428] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:07.321597 24856 raft_consensus.cc:348] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.321658 24856 raft_consensus.cc:374] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.321628 24869 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.321686 24856 raft_consensus.cc:729] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49207139453e4953a409a65efe167400, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.321704 24869 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:07.321725 24856 consensus_queue.cc:260] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.321810 24856 ts_tablet_manager.cc:1428] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:07.321841 24870 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.321908 24870 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 06:57:07.352073 24821 tablet.cc:2369] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:07.362717 24869 raft_consensus.cc:482] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:07.362782 24869 raft_consensus.cc:504] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.363124 24869 leader_election.cc:288] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49207139453e4953a409a65efe167400 (127.21.28.65:37195), 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:07.366725 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.363298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:53521 (local address 127.21.28.66:45525)
0808 06:57:07.363410 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.363412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.363485 (+ 73us) server_negotiation.cc:407] Connection header received
0808 06:57:07.363602 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.363603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.363612 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.363653 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:07.363798 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.364489 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.365839 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.366375 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.366445 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.366455 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.366621 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.366659 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 06:57:07.366726 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.363298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:60363 (local address 127.21.28.65:37195)
0808 06:57:07.363338 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.363340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.363351 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:57:07.363444 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.363445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.363454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.363499 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:07.363708 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.364459 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.365840 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.366403 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.366461 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.366468 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.366621 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.366659 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:07.366746 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.363247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:37195 (local address 127.21.28.67:60363)
0808 06:57:07.363316 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.363327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.363365 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.363548 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.363548 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.363550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:07.363619 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.363621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.364530 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.364530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.365761 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.365764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.366466 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.366466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.366551 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.366556 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.366611 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.366659 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":72}
I20260808 06:57:07.366756 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.363257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:45525 (local address 127.21.28.67:53521)
0808 06:57:07.363417 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.363428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.363511 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.363656 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.363656 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.363658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:07.363746 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.363747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.364546 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.364547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.365761 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.365764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.366462 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.366463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.366550 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.366555 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.366611 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.366657 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":12,"thread_start_us":102,"threads_started":1}
I20260808 06:57:07.367031 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cce9c8f39045f1be85b6888d1fe06d" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a918e2e600d44268412dc6196c1f777" is_pre_election: true
I20260808 06:57:07.367046 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cce9c8f39045f1be85b6888d1fe06d" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49207139453e4953a409a65efe167400" is_pre_election: true
I20260808 06:57:07.367116 24551 raft_consensus.cc:2454] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 0.
I20260808 06:57:07.367142 24352 raft_consensus.cc:2454] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 0.
I20260808 06:57:07.367324 24652 leader_election.cc:302] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487586eb30df460e9d3416738c9f1314, 49207139453e4953a409a65efe167400; no voters:
I20260808 06:57:07.367456 24869 raft_consensus.cc:2781] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:07.367502 24869 raft_consensus.cc:482] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:07.367517 24869 raft_consensus.cc:3037] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.368209 24869 raft_consensus.cc:504] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.368319 24869 leader_election.cc:288] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 election: Requested vote from peers 49207139453e4953a409a65efe167400 (127.21.28.65:37195), 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:07.368494 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cce9c8f39045f1be85b6888d1fe06d" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a918e2e600d44268412dc6196c1f777"
I20260808 06:57:07.368510 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cce9c8f39045f1be85b6888d1fe06d" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49207139453e4953a409a65efe167400"
I20260808 06:57:07.368583 24551 raft_consensus.cc:3037] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.368582 24352 raft_consensus.cc:3037] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.369364 24352 raft_consensus.cc:2454] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 1.
I20260808 06:57:07.369364 24551 raft_consensus.cc:2454] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 1.
I20260808 06:57:07.369540 24656 leader_election.cc:302] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a918e2e600d44268412dc6196c1f777, 487586eb30df460e9d3416738c9f1314; no voters:
I20260808 06:57:07.369650 24869 raft_consensus.cc:2781] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:07.369800 24869 raft_consensus.cc:686] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [term 1 LEADER]: Becoming Leader. State: Replica: 487586eb30df460e9d3416738c9f1314, State: Running, Role: LEADER
I20260808 06:57:07.369892 24869 consensus_queue.cc:237] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.370350 24890 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: TxnStatusTablet state changed. Reason: New leader 487586eb30df460e9d3416738c9f1314. Latest consensus state: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.370440 24890 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:07.370615 24894 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 06:57:07.370728 24894 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 06:57:07.370664 24137 catalog_manager.cc:5495] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 reported cstate change: term changed from 0 to 1, leader changed from <none> to 487586eb30df460e9d3416738c9f1314 (127.21.28.67). New cstate: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } health_report { overall_health: HEALTHY } } }
I20260808 06:57:07.379627 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.377391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.21.28.67:42069)
0808 06:57:07.377429 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.377431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.377464 (+ 33us) server_negotiation.cc:407] Connection header received
0808 06:57:07.377543 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.377544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.377560 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.377638 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:07.377735 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.378197 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.378900 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.379099 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.379151 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.379342 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.379434 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.379499 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.379540 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.379563 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:07.379652 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.377348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:42069 (local address 127.0.0.1:50214)
0808 06:57:07.377424 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.377435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.377483 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.377643 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.377644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.377645 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:07.377689 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.377691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.378268 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.378269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.378829 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.378831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.379155 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.379156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.379245 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.379261 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.379488 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.379496 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.379538 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.379588 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 06:57:07.381198 24352 raft_consensus.cc:1261] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Refusing update from remote peer 487586eb30df460e9d3416738c9f1314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:07.381251 24551 raft_consensus.cc:1261] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Refusing update from remote peer 487586eb30df460e9d3416738c9f1314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:07.381492 24890 consensus_queue.cc:1035] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:07.381614 24869 consensus_queue.cc:1035] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:07.382774 24865 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: TxnStatusTablet state changed. Reason: New leader 487586eb30df460e9d3416738c9f1314. Latest consensus state: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.382874 24865 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:07.382938 24870 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: TxnStatusTablet state changed. Reason: New leader 487586eb30df460e9d3416738c9f1314. Latest consensus state: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.383014 24870 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:07.383837 24869 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.383911 24890 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.383988 24890 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:07.384299 24900 mvcc.cc:205] Tried to move back new op lower bound from 7316161443351195648 to 7316161443307716608. Current Snapshot: MvccSnapshot[applied={T|T < 7316161443351195648}]
I20260808 06:57:07.384414 24870 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.384492 24870 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 49207139453e4953a409a65efe167400: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:07.384554 24865 tablet_replica.cc:439] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } } }
I20260808 06:57:07.384625 24865 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 1a918e2e600d44268412dc6196c1f777: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:07.383921 24869 tablet_replica.cc:442] T e6cce9c8f39045f1be85b6888d1fe06d P 487586eb30df460e9d3416738c9f1314: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:07.385162 24901 mvcc.cc:205] Tried to move back new op lower bound from 7316161443351195648 to 7316161443307716608. Current Snapshot: MvccSnapshot[applied={T|T < 7316161443351195648}]
I20260808 06:57:07.389684 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.387526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.21.28.65:37195)
0808 06:57:07.387571 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.387573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.387579 (+ 6us) server_negotiation.cc:407] Connection header received
0808 06:57:07.387607 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.387608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.387622 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.387692 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:07.387836 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.388285 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.389002 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.389201 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.389260 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.389394 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.389498 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.389563 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.389600 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.389622 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 06:57:07.394785 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.392586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:50069 (local address 127.21.28.67:42069)
0808 06:57:07.392619 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.392621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.392761 (+ 140us) server_negotiation.cc:407] Connection header received
0808 06:57:07.392815 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.392816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.392826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.392866 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:07.392992 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.393441 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.394189 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.394368 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.394421 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.394587 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.394618 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.394657 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.394710 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.394726 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 06:57:07.394833 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.392546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:42069 (local address 127.21.28.65:50069)
0808 06:57:07.392700 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.392709 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.392762 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.392873 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.392874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.392875 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:07.392938 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.392939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.393509 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.393510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.394117 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.394119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.394434 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.394434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.394498 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.394511 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.394669 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.394670 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.394709 (+ 39us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.394750 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260808 06:57:07.397230 24551 raft_consensus.cc:1261] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Refusing update from remote peer 49207139453e4953a409a65efe167400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:07.397253 24751 raft_consensus.cc:1261] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Refusing update from remote peer 49207139453e4953a409a65efe167400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:07.397480 24870 consensus_queue.cc:1035] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:07.397612 24872 consensus_queue.cc:1035] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:07.399827 24899 mvcc.cc:205] Tried to move back new op lower bound from 7316161443417391104 to 7316161442898661376. Current Snapshot: MvccSnapshot[applied={T|T < 7316161443417391104}]
I20260808 06:57:07.414556 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.411980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:37195 (local address 127.21.28.67:42197)
0808 06:57:07.412070 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.412080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.412122 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.412265 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.412266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.412267 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:07.412329 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.412330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.413191 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.413192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.413779 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.413780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.414238 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.414238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.414287 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.414300 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.414446 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.414447 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.414472 (+ 25us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.414506 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 06:57:07.414578 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.412023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:42197 (local address 127.21.28.65:37195)
0808 06:57:07.412057 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.412059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.412133 (+ 74us) server_negotiation.cc:407] Connection header received
0808 06:57:07.412159 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.412160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.412170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.412220 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:07.412394 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.413120 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.413855 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.414163 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.414223 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.414364 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.414395 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.414437 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.414489 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.414508 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 06:57:07.422220 24137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:07.422322 24137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:07.423012 24137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: Sending DeleteTablet for 3 replicas of tablet 1b3ba11791224c16838a36f2b6b88de7
I20260808 06:57:07.423413 24521 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3ba11791224c16838a36f2b6b88de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:07 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:54550
I20260808 06:57:07.423632 24329 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3ba11791224c16838a36f2b6b88de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:07 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:44742
I20260808 06:57:07.423691 24724 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3ba11791224c16838a36f2b6b88de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:07 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:50200
I20260808 06:57:07.423878 24137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 06:57:07.423942 24930 tablet_replica.cc:330] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: stopping tablet replica
W20260808 06:57:07.424016 24137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:07.424196 24930 raft_consensus.cc:2229] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:07.424341 24930 raft_consensus.cc:2258] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:07.424808 24930 ts_tablet_manager.cc:1905] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:07.425369 24521 tablet_service.cc:1467] Processing CreateTablet for tablet a13f686e1592446e872d1bba633956ed (DEFAULT_TABLE table=test-table [id=8746c64685084a0fb50d6a30987ff836]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:07.425595 24521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13f686e1592446e872d1bba633956ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.426764 24930 ts_tablet_manager.cc:1918] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:07.426913 24930 log.cc:1196] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/1b3ba11791224c16838a36f2b6b88de7
I20260808 06:57:07.427099 24858 tablet_bootstrap.cc:492] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: Bootstrap starting.
I20260808 06:57:07.427296 24930 ts_tablet_manager.cc:1939] T 1b3ba11791224c16838a36f2b6b88de7 P 49207139453e4953a409a65efe167400: Deleting consensus metadata
I20260808 06:57:07.427337 24329 tablet_service.cc:1467] Processing CreateTablet for tablet a13f686e1592446e872d1bba633956ed (DEFAULT_TABLE table=test-table [id=8746c64685084a0fb50d6a30987ff836]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:07.427450 24329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13f686e1592446e872d1bba633956ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.427474 24724 tablet_service.cc:1467] Processing CreateTablet for tablet a13f686e1592446e872d1bba633956ed (DEFAULT_TABLE table=test-table [id=8746c64685084a0fb50d6a30987ff836]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:07.427508 24858 tablet_bootstrap.cc:654] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.427578 24724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13f686e1592446e872d1bba633956ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:07.428638 24856 tablet_bootstrap.cc:492] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: Bootstrap starting.
I20260808 06:57:07.429018 24856 tablet_bootstrap.cc:654] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.429065 24851 tablet_bootstrap.cc:492] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: Bootstrap starting.
I20260808 06:57:07.429427 24851 tablet_bootstrap.cc:654] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:07.429888 24119 catalog_manager.cc:4841] TS 49207139453e4953a409a65efe167400 (127.21.28.65:37195): tablet 1b3ba11791224c16838a36f2b6b88de7 (table test-table [id=56222b0734e64bd09553466a97d9686e]) successfully deleted
I20260808 06:57:07.430332 24856 tablet_bootstrap.cc:492] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: No bootstrap required, opened a new log
I20260808 06:57:07.430377 24856 ts_tablet_manager.cc:1397] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 06:57:07.430501 24856 raft_consensus.cc:348] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.430558 24856 raft_consensus.cc:374] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.430576 24856 raft_consensus.cc:729] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49207139453e4953a409a65efe167400, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.430612 24856 consensus_queue.cc:260] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.430627 24851 tablet_bootstrap.cc:492] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: No bootstrap required, opened a new log
I20260808 06:57:07.430673 24851 ts_tablet_manager.cc:1397] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:07.430682 24856 ts_tablet_manager.cc:1428] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:07.430837 24851 raft_consensus.cc:348] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.430907 24851 raft_consensus.cc:374] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.430927 24851 raft_consensus.cc:729] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487586eb30df460e9d3416738c9f1314, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.430963 24851 consensus_queue.cc:260] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.431061 24851 ts_tablet_manager.cc:1428] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:07.431226 24927 tablet_replica.cc:330] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: stopping tablet replica
I20260808 06:57:07.431324 24927 raft_consensus.cc:2229] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:07.431373 24927 raft_consensus.cc:2258] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:07.431548 24931 tablet_replica.cc:330] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: stopping tablet replica
I20260808 06:57:07.431767 24931 raft_consensus.cc:2229] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:07.431814 24931 raft_consensus.cc:2258] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:07.432201 24931 ts_tablet_manager.cc:1905] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:07.432683 24927 ts_tablet_manager.cc:1905] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:07.433717 24931 ts_tablet_manager.cc:1918] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:07.433789 24931 log.cc:1196] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/1b3ba11791224c16838a36f2b6b88de7
I20260808 06:57:07.434046 24931 ts_tablet_manager.cc:1939] T 1b3ba11791224c16838a36f2b6b88de7 P 487586eb30df460e9d3416738c9f1314: Deleting consensus metadata
I20260808 06:57:07.434420 24119 catalog_manager.cc:4841] TS 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069): tablet 1b3ba11791224c16838a36f2b6b88de7 (table test-table [id=56222b0734e64bd09553466a97d9686e]) successfully deleted
I20260808 06:57:07.434659 24927 ts_tablet_manager.cc:1918] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:07.434765 24927 log.cc:1196] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/1b3ba11791224c16838a36f2b6b88de7
I20260808 06:57:07.435035 24927 ts_tablet_manager.cc:1939] T 1b3ba11791224c16838a36f2b6b88de7 P 1a918e2e600d44268412dc6196c1f777: Deleting consensus metadata
I20260808 06:57:07.435489 24121 catalog_manager.cc:4841] TS 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525): tablet 1b3ba11791224c16838a36f2b6b88de7 (table test-table [id=56222b0734e64bd09553466a97d9686e]) successfully deleted
I20260808 06:57:07.443030 24858 tablet_bootstrap.cc:492] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: No bootstrap required, opened a new log
I20260808 06:57:07.443092 24858 ts_tablet_manager.cc:1397] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20260808 06:57:07.443248 24858 raft_consensus.cc:348] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.443325 24858 raft_consensus.cc:374] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:07.443344 24858 raft_consensus.cc:729] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a918e2e600d44268412dc6196c1f777, State: Initialized, Role: FOLLOWER
I20260808 06:57:07.443390 24858 consensus_queue.cc:260] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.443478 24858 ts_tablet_manager.cc:1428] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:07.451370 24890 raft_consensus.cc:482] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:07.451450 24890 raft_consensus.cc:504] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.451587 24890 leader_election.cc:288] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49207139453e4953a409a65efe167400 (127.21.28.65:37195), 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:07.451759 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13f686e1592446e872d1bba633956ed" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49207139453e4953a409a65efe167400" is_pre_election: true
I20260808 06:57:07.451846 24352 raft_consensus.cc:2454] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 0.
I20260808 06:57:07.451798 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13f686e1592446e872d1bba633956ed" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a918e2e600d44268412dc6196c1f777" is_pre_election: true
I20260808 06:57:07.451892 24551 raft_consensus.cc:2454] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 0.
I20260808 06:57:07.451996 24652 leader_election.cc:302] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487586eb30df460e9d3416738c9f1314, 49207139453e4953a409a65efe167400; no voters:
I20260808 06:57:07.452263 24890 raft_consensus.cc:2781] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:07.452303 24890 raft_consensus.cc:482] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:07.452323 24890 raft_consensus.cc:3037] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.453150 24890 raft_consensus.cc:504] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.453271 24890 leader_election.cc:288] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 election: Requested vote from peers 49207139453e4953a409a65efe167400 (127.21.28.65:37195), 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:07.453430 24551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13f686e1592446e872d1bba633956ed" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a918e2e600d44268412dc6196c1f777"
I20260808 06:57:07.453503 24551 raft_consensus.cc:3037] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.454229 24551 raft_consensus.cc:2454] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 1.
I20260808 06:57:07.454382 24656 leader_election.cc:302] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a918e2e600d44268412dc6196c1f777, 487586eb30df460e9d3416738c9f1314; no voters:
I20260808 06:57:07.454516 24890 raft_consensus.cc:2781] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:07.454504 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13f686e1592446e872d1bba633956ed" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49207139453e4953a409a65efe167400"
I20260808 06:57:07.454561 24890 raft_consensus.cc:686] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 1 LEADER]: Becoming Leader. State: Replica: 487586eb30df460e9d3416738c9f1314, State: Running, Role: LEADER
I20260808 06:57:07.454576 24352 raft_consensus.cc:3037] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:07.454602 24890 consensus_queue.cc:237] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:07.455977 24352 raft_consensus.cc:2454] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 1.
I20260808 06:57:07.456203 24137 catalog_manager.cc:5495] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 reported cstate change: term changed from 0 to 1, leader changed from <none> to 487586eb30df460e9d3416738c9f1314 (127.21.28.67). New cstate: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } health_report { overall_health: HEALTHY } } }
I20260808 06:57:07.464783 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.461918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.21.28.67:42069)
0808 06:57:07.461955 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.461957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.461966 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:07.462019 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.462020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.462030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.462094 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:07.462202 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.462966 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.464045 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.464335 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.464409 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.464550 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.464587 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.464641 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.464692 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.464711 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:57:07.469770 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.467033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:55283 (local address 127.21.28.67:42069)
0808 06:57:07.467085 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:07.467087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:07.467094 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:07.467113 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:07.467114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:07.467122 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:07.467176 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:07.467290 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.468001 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.469019 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.469314 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.469378 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.469532 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.469561 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.469614 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:07.469664 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 06:57:07.469680 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:57:07.469826 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:07.466849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:42069 (local address 127.21.28.67:55283)
0808 06:57:07.466898 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:07.466910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:07.466953 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:07.467187 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:07.467188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:07.467189 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:07.467227 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.467228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.468075 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.468076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.468930 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:07.468932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:07.469400 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:07.469400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:07.469465 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:07.469469 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:07.469618 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:07.469620 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:07.469661 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 06:57:07.469703 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 06:57:07.472091 24352 raft_consensus.cc:1261] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Refusing update from remote peer 487586eb30df460e9d3416738c9f1314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:07.472177 24551 raft_consensus.cc:1261] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Refusing update from remote peer 487586eb30df460e9d3416738c9f1314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:07.472396 24906 consensus_queue.cc:1035] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:07.472534 24869 consensus_queue.cc:1035] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 06:57:07.480309 24899 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 06:57:08.998293 24136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:08.998392 24136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:08.999153 24136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: Sending DeleteTablet for 3 replicas of tablet a13f686e1592446e872d1bba633956ed
I20260808 06:57:08.999403 24329 tablet_service.cc:1514] Processing DeleteTablet for tablet a13f686e1592446e872d1bba633956ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:08 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:44742
I20260808 06:57:08.999594 24521 tablet_service.cc:1514] Processing DeleteTablet for tablet a13f686e1592446e872d1bba633956ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:08 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:54550
I20260808 06:57:09.001173 24722 tablet_service.cc:1514] Processing DeleteTablet for tablet a13f686e1592446e872d1bba633956ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:08 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:50200
I20260808 06:57:09.002210 24966 tablet_replica.cc:330] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: stopping tablet replica
I20260808 06:57:09.002285 24966 raft_consensus.cc:2229] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:09.002323 24966 raft_consensus.cc:2258] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:09.002424 24967 tablet_replica.cc:330] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: stopping tablet replica
I20260808 06:57:09.002473 24967 raft_consensus.cc:2229] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:09.002501 24967 raft_consensus.cc:2258] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:09.002584 24970 tablet_replica.cc:330] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: stopping tablet replica
I20260808 06:57:09.002641 24970 raft_consensus.cc:2229] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:09.002719 24970 raft_consensus.cc:2258] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:09.002724 24967 ts_tablet_manager.cc:1905] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:09.003850 24970 ts_tablet_manager.cc:1905] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:09.004135 24966 ts_tablet_manager.cc:1905] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:09.004281 24967 ts_tablet_manager.cc:1918] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 06:57:09.004340 24967 log.cc:1196] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/a13f686e1592446e872d1bba633956ed
I20260808 06:57:09.004572 24967 ts_tablet_manager.cc:1939] T a13f686e1592446e872d1bba633956ed P 1a918e2e600d44268412dc6196c1f777: Deleting consensus metadata
I20260808 06:57:09.004882 24121 catalog_manager.cc:4841] TS 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525): tablet a13f686e1592446e872d1bba633956ed (table test-table [id=8746c64685084a0fb50d6a30987ff836]) successfully deleted
I20260808 06:57:09.005761 24970 ts_tablet_manager.cc:1918] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 06:57:09.005818 24970 log.cc:1196] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/a13f686e1592446e872d1bba633956ed
I20260808 06:57:09.006000 24966 ts_tablet_manager.cc:1918] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 06:57:09.006042 24966 log.cc:1196] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/a13f686e1592446e872d1bba633956ed
I20260808 06:57:09.006060 24970 ts_tablet_manager.cc:1939] T a13f686e1592446e872d1bba633956ed P 487586eb30df460e9d3416738c9f1314: Deleting consensus metadata
I20260808 06:57:09.006214 24966 ts_tablet_manager.cc:1939] T a13f686e1592446e872d1bba633956ed P 49207139453e4953a409a65efe167400: Deleting consensus metadata
I20260808 06:57:09.006346 24119 catalog_manager.cc:4841] TS 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069): tablet a13f686e1592446e872d1bba633956ed (table test-table [id=8746c64685084a0fb50d6a30987ff836]) successfully deleted
I20260808 06:57:09.006563 24119 catalog_manager.cc:4841] TS 49207139453e4953a409a65efe167400 (127.21.28.65:37195): tablet a13f686e1592446e872d1bba633956ed (table test-table [id=8746c64685084a0fb50d6a30987ff836]) successfully deleted
Aug 08 06:57:09 dist-test-slave-nj21 krb5kdc[24083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 06:57:09 dist-test-slave-nj21 krb5kdc[24083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172229, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:09.022353 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.016246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.21.28.126:33957)
0808 06:57:09.016363 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:09.016365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:09.016373 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:09.016442 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:09.016443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:09.016452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:09.016506 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:09.017011 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.017507 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.018207 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.018401 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.018450 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.020414 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:09.020415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:09.020415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:09.020421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:09.021567 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:09.021829 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:09.021830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:09.021830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:09.021874 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:09.022000 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:09.022000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:09.022000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:09.022089 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:09.022155 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:09.022224 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 06:57:09.022282 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
I20260808 06:57:09.023535 24136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:09.023722 24136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:09.024911 24329 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf507ab15ab4624aeba947dc4412c24 (DEFAULT_TABLE table=test-table [id=0e80277b1d4e4ac0a2ec84aa5066fb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:09.024937 24722 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf507ab15ab4624aeba947dc4412c24 (DEFAULT_TABLE table=test-table [id=0e80277b1d4e4ac0a2ec84aa5066fb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:09.025024 24329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf507ab15ab4624aeba947dc4412c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:09.025050 24722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf507ab15ab4624aeba947dc4412c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:09.025926 24984 tablet_bootstrap.cc:492] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: Bootstrap starting.
I20260808 06:57:09.025969 24521 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf507ab15ab4624aeba947dc4412c24 (DEFAULT_TABLE table=test-table [id=0e80277b1d4e4ac0a2ec84aa5066fb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:09.026069 24521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf507ab15ab4624aeba947dc4412c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:09.026307 24984 tablet_bootstrap.cc:654] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:09.027011 24984 tablet_bootstrap.cc:492] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: No bootstrap required, opened a new log
I20260808 06:57:09.027066 24984 ts_tablet_manager.cc:1397] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:09.027184 24984 raft_consensus.cc:348] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.027257 24984 raft_consensus.cc:374] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:09.027277 24984 raft_consensus.cc:729] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49207139453e4953a409a65efe167400, State: Initialized, Role: FOLLOWER
I20260808 06:57:09.027282 24989 tablet_bootstrap.cc:492] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: Bootstrap starting.
I20260808 06:57:09.027318 24985 tablet_bootstrap.cc:492] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: Bootstrap starting.
I20260808 06:57:09.027339 24984 consensus_queue.cc:260] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.027513 24984 ts_tablet_manager.cc:1428] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:09.027659 24989 tablet_bootstrap.cc:654] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:09.027726 24985 tablet_bootstrap.cc:654] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:09.028256 24985 tablet_bootstrap.cc:492] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: No bootstrap required, opened a new log
I20260808 06:57:09.028308 24985 ts_tablet_manager.cc:1397] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:09.028326 24989 tablet_bootstrap.cc:492] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: No bootstrap required, opened a new log
I20260808 06:57:09.028422 24989 ts_tablet_manager.cc:1397] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:09.028416 24985 raft_consensus.cc:348] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.028487 24985 raft_consensus.cc:374] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:09.028517 24985 raft_consensus.cc:729] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487586eb30df460e9d3416738c9f1314, State: Initialized, Role: FOLLOWER
I20260808 06:57:09.028560 24985 consensus_queue.cc:260] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.028556 24989 raft_consensus.cc:348] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.028618 24989 raft_consensus.cc:374] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:09.028637 24989 raft_consensus.cc:729] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a918e2e600d44268412dc6196c1f777, State: Initialized, Role: FOLLOWER
I20260808 06:57:09.028651 24985 ts_tablet_manager.cc:1428] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:09.028772 24989 consensus_queue.cc:260] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.028949 24989 ts_tablet_manager.cc:1428] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:09.040010 24995 raft_consensus.cc:482] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:09.040073 24995 raft_consensus.cc:504] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.040321 24995 leader_election.cc:288] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49207139453e4953a409a65efe167400 (127.21.28.65:37195), 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069)
I20260808 06:57:09.043254 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.040721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:59419 (local address 127.21.28.67:42069)
0808 06:57:09.040839 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:09.040841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:09.040848 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:09.040868 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:09.040868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:09.040879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:09.040962 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:09.041159 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.041680 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.042548 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.042950 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.043005 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.043015 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:09.043143 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 06:57:09.043184 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260808 06:57:09.043278 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.040513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:40873 (local address 127.21.28.65:37195)
0808 06:57:09.040662 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:09.040664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:09.040672 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:09.040712 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:09.040712 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:09.040722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:09.040786 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:09.040976 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.041535 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.042524 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.042962 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.043029 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.043037 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:09.043169 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 06:57:09.043204 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260808 06:57:09.043278 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.040656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:42069 (local address 127.21.28.66:59419)
0808 06:57:09.040760 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:09.040768 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:09.040805 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:09.041029 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:09.041029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:09.041032 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:09.041099 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:09.041101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.041752 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.041753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:09.042469 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:09.042471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.043020 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.043021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:09.043086 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.043090 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:09.043129 (+ 39us) client_negotiation.cc:240] Negotiation successful
0808 06:57:09.043169 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":68,"threads_started":1}
I20260808 06:57:09.043287 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.040452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:37195 (local address 127.21.28.66:40873)
0808 06:57:09.040596 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:09.040608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:09.040664 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:09.040793 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:09.040793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:09.040796 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:09.040864 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:09.040867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.041605 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.041606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:09.042445 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:09.042447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.043036 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.043037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:09.043106 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.043109 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:09.043158 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 06:57:09.043194 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":96,"threads_started":1}
I20260808 06:57:09.043607 24751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf507ab15ab4624aeba947dc4412c24" candidate_uuid: "1a918e2e600d44268412dc6196c1f777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487586eb30df460e9d3416738c9f1314" is_pre_election: true
I20260808 06:57:09.043620 24351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf507ab15ab4624aeba947dc4412c24" candidate_uuid: "1a918e2e600d44268412dc6196c1f777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49207139453e4953a409a65efe167400" is_pre_election: true
I20260808 06:57:09.043691 24751 raft_consensus.cc:2454] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a918e2e600d44268412dc6196c1f777 in term 0.
I20260808 06:57:09.043715 24351 raft_consensus.cc:2454] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a918e2e600d44268412dc6196c1f777 in term 0.
I20260808 06:57:09.043884 24456 leader_election.cc:302] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a918e2e600d44268412dc6196c1f777, 49207139453e4953a409a65efe167400; no voters:
I20260808 06:57:09.044005 24995 raft_consensus.cc:2781] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:09.044044 24995 raft_consensus.cc:482] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:09.044070 24995 raft_consensus.cc:3037] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:09.044641 24995 raft_consensus.cc:504] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.044734 24995 leader_election.cc:288] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [CANDIDATE]: Term 1 election: Requested vote from peers 49207139453e4953a409a65efe167400 (127.21.28.65:37195), 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069)
I20260808 06:57:09.044888 24351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf507ab15ab4624aeba947dc4412c24" candidate_uuid: "1a918e2e600d44268412dc6196c1f777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49207139453e4953a409a65efe167400"
I20260808 06:57:09.044905 24751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf507ab15ab4624aeba947dc4412c24" candidate_uuid: "1a918e2e600d44268412dc6196c1f777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487586eb30df460e9d3416738c9f1314"
I20260808 06:57:09.044950 24351 raft_consensus.cc:3037] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:09.044981 24751 raft_consensus.cc:3037] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:09.045485 24351 raft_consensus.cc:2454] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a918e2e600d44268412dc6196c1f777 in term 1.
I20260808 06:57:09.045598 24751 raft_consensus.cc:2454] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a918e2e600d44268412dc6196c1f777 in term 1.
I20260808 06:57:09.045642 24456 leader_election.cc:302] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a918e2e600d44268412dc6196c1f777, 49207139453e4953a409a65efe167400; no voters:
I20260808 06:57:09.045710 24995 raft_consensus.cc:2781] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:09.045838 24995 raft_consensus.cc:686] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 1 LEADER]: Becoming Leader. State: Replica: 1a918e2e600d44268412dc6196c1f777, State: Running, Role: LEADER
I20260808 06:57:09.045895 24995 consensus_queue.cc:237] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.046463 24137 catalog_manager.cc:5495] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a918e2e600d44268412dc6196c1f777 (127.21.28.66). New cstate: current_term: 1 leader_uuid: "1a918e2e600d44268412dc6196c1f777" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } health_report { overall_health: UNKNOWN } } }
I20260808 06:57:09.057718 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.055413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.21.28.66:45525)
0808 06:57:09.055543 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:09.055545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:09.055552 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:09.055579 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:09.055579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:09.055595 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:09.055660 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:09.055808 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.056269 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.056967 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.057181 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.057232 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.057392 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:09.057497 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:09.057558 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:09.057607 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 06:57:09.057624 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260808 06:57:09.061939 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.059803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:57477 (local address 127.21.28.67:42069)
0808 06:57:09.059850 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:09.059852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:09.059917 (+ 65us) server_negotiation.cc:407] Connection header received
0808 06:57:09.059974 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:09.059974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:09.059985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:09.060029 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:09.060149 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.060621 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.061334 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.061533 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.061582 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.061735 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:09.061767 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:09.061808 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:09.061858 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 06:57:09.061876 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 06:57:09.061970 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.059655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:42069 (local address 127.21.28.66:57477)
0808 06:57:09.059857 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:09.059867 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:09.059931 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:09.060039 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:09.060039 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:09.060041 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:09.060098 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:09.060099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.060688 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.060689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:09.061266 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:09.061267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.061594 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.061595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:09.061660 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.061672 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:09.061816 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:09.061818 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:09.061857 (+ 39us) client_negotiation.cc:240] Negotiation successful
0808 06:57:09.061890 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":114,"threads_started":1}
I20260808 06:57:09.063906 24751 raft_consensus.cc:1261] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Refusing update from remote peer 1a918e2e600d44268412dc6196c1f777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:09.063938 24351 raft_consensus.cc:1261] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Refusing update from remote peer 1a918e2e600d44268412dc6196c1f777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:09.064177 24995 consensus_queue.cc:1035] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:09.064312 25002 consensus_queue.cc:1035] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:09.070631 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.068485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:60727 (local address 127.21.28.66:45525)
0808 06:57:09.068517 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:09.068519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:09.068619 (+ 100us) server_negotiation.cc:407] Connection header received
0808 06:57:09.068668 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:09.068668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:09.068676 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:09.068711 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:09.068803 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.069231 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.069954 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.070167 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.070218 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.070416 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:09.070445 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:09.070487 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:09.070537 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 06:57:09.070554 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 06:57:09.070641 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.068435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:45525 (local address 127.21.28.67:60727)
0808 06:57:09.068558 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:09.068567 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:09.068622 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:09.068730 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:09.068731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:09.068732 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:09.068765 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:09.068766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.069318 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.069319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:09.069892 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:09.069893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.070242 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.070243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:09.070301 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.070305 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:09.070487 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:09.070489 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:09.070526 (+ 37us) client_negotiation.cc:240] Negotiation successful
0808 06:57:09.070569 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":78,"threads_started":1}
I20260808 06:57:09.077073 24136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:09.077168 24136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:09.077836 24136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: Sending DeleteTablet for 3 replicas of tablet 2bf507ab15ab4624aeba947dc4412c24
I20260808 06:57:09.078047 24521 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf507ab15ab4624aeba947dc4412c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:09 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:54550
I20260808 06:57:09.078063 24722 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf507ab15ab4624aeba947dc4412c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:09 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:50200
I20260808 06:57:09.078135 24967 tablet_replica.cc:330] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: stopping tablet replica
I20260808 06:57:09.078114 24329 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf507ab15ab4624aeba947dc4412c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:09 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:44742
I20260808 06:57:09.078147 24970 tablet_replica.cc:330] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: stopping tablet replica
I20260808 06:57:09.078176 24967 raft_consensus.cc:2229] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:09.078193 24970 raft_consensus.cc:2229] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:09.078222 24970 raft_consensus.cc:2258] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:09.078271 24967 raft_consensus.cc:2258] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:09.078293 24966 tablet_replica.cc:330] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: stopping tablet replica
I20260808 06:57:09.078361 24966 raft_consensus.cc:2229] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:09.078404 24966 raft_consensus.cc:2258] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:09.078559 24967 ts_tablet_manager.cc:1905] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:09.078668 24970 ts_tablet_manager.cc:1905] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:09.078800 24136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:09.078919 24136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:09.080031 24521 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019 (DEFAULT_TABLE table=test-table [id=74d6642ef3414302b1669bddc795e19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:09.080176 24521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:09.080196 24967 ts_tablet_manager.cc:1918] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:09.080248 24967 log.cc:1196] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/2bf507ab15ab4624aeba947dc4412c24
I20260808 06:57:09.080461 24967 ts_tablet_manager.cc:1939] T 2bf507ab15ab4624aeba947dc4412c24 P 1a918e2e600d44268412dc6196c1f777: Deleting consensus metadata
I20260808 06:57:09.080521 24966 ts_tablet_manager.cc:1905] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:09.080617 24970 ts_tablet_manager.cc:1918] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:09.080667 24970 log.cc:1196] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/2bf507ab15ab4624aeba947dc4412c24
I20260808 06:57:09.080866 24970 ts_tablet_manager.cc:1939] T 2bf507ab15ab4624aeba947dc4412c24 P 487586eb30df460e9d3416738c9f1314: Deleting consensus metadata
I20260808 06:57:09.080859 24121 catalog_manager.cc:4841] TS 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525): tablet 2bf507ab15ab4624aeba947dc4412c24 (table test-table [id=0e80277b1d4e4ac0a2ec84aa5066fb05]) successfully deleted
I20260808 06:57:09.080971 24722 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019 (DEFAULT_TABLE table=test-table [id=74d6642ef3414302b1669bddc795e19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:09.081065 24722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:09.081292 24989 tablet_bootstrap.cc:492] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: Bootstrap starting.
I20260808 06:57:09.081666 24989 tablet_bootstrap.cc:654] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:09.082018 24966 ts_tablet_manager.cc:1918] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:09.082080 24966 log.cc:1196] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/2bf507ab15ab4624aeba947dc4412c24
I20260808 06:57:09.082267 24985 tablet_bootstrap.cc:492] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: Bootstrap starting.
I20260808 06:57:09.082290 24966 ts_tablet_manager.cc:1939] T 2bf507ab15ab4624aeba947dc4412c24 P 49207139453e4953a409a65efe167400: Deleting consensus metadata
I20260808 06:57:09.082341 24119 catalog_manager.cc:4841] TS 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069): tablet 2bf507ab15ab4624aeba947dc4412c24 (table test-table [id=0e80277b1d4e4ac0a2ec84aa5066fb05]) successfully deleted
I20260808 06:57:09.082517 24329 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019 (DEFAULT_TABLE table=test-table [id=74d6642ef3414302b1669bddc795e19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:09.082599 24119 catalog_manager.cc:4841] TS 49207139453e4953a409a65efe167400 (127.21.28.65:37195): tablet 2bf507ab15ab4624aeba947dc4412c24 (table test-table [id=0e80277b1d4e4ac0a2ec84aa5066fb05]) successfully deleted
I20260808 06:57:09.082612 24329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:09.082607 24985 tablet_bootstrap.cc:654] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:09.083166 24989 tablet_bootstrap.cc:492] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: No bootstrap required, opened a new log
I20260808 06:57:09.083214 24989 ts_tablet_manager.cc:1397] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 06:57:09.083366 24989 raft_consensus.cc:348] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.083425 24989 raft_consensus.cc:374] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:09.083444 24989 raft_consensus.cc:729] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a918e2e600d44268412dc6196c1f777, State: Initialized, Role: FOLLOWER
I20260808 06:57:09.083483 24989 consensus_queue.cc:260] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.083559 24984 tablet_bootstrap.cc:492] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400: Bootstrap starting.
I20260808 06:57:09.083578 24989 ts_tablet_manager.cc:1428] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:09.083863 24984 tablet_bootstrap.cc:654] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:09.083976 24985 tablet_bootstrap.cc:492] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: No bootstrap required, opened a new log
I20260808 06:57:09.084020 24985 ts_tablet_manager.cc:1397] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 06:57:09.084128 24985 raft_consensus.cc:348] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.084183 24985 raft_consensus.cc:374] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:09.084203 24985 raft_consensus.cc:729] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487586eb30df460e9d3416738c9f1314, State: Initialized, Role: FOLLOWER
I20260808 06:57:09.084242 24985 consensus_queue.cc:260] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.084333 24985 ts_tablet_manager.cc:1428] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:09.084334 24984 tablet_bootstrap.cc:492] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400: No bootstrap required, opened a new log
I20260808 06:57:09.084376 24984 ts_tablet_manager.cc:1397] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:09.084465 24984 raft_consensus.cc:348] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.084545 24984 raft_consensus.cc:374] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:09.084576 24984 raft_consensus.cc:729] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49207139453e4953a409a65efe167400, State: Initialized, Role: FOLLOWER
I20260808 06:57:09.084621 24984 consensus_queue.cc:260] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.084728 24984 ts_tablet_manager.cc:1428] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:09.106648 24960 raft_consensus.cc:482] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:09.106724 24960 raft_consensus.cc:504] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.106869 24960 leader_election.cc:288] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49207139453e4953a409a65efe167400 (127.21.28.65:37195), 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:09.107053 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3cf29e3adb4c86a6176e3a1aeb5019" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49207139453e4953a409a65efe167400" is_pre_election: true
I20260808 06:57:09.107142 24352 raft_consensus.cc:2454] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 0.
I20260808 06:57:09.107121 24550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3cf29e3adb4c86a6176e3a1aeb5019" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a918e2e600d44268412dc6196c1f777" is_pre_election: true
I20260808 06:57:09.107228 24550 raft_consensus.cc:2454] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 0.
I20260808 06:57:09.107292 24652 leader_election.cc:302] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487586eb30df460e9d3416738c9f1314, 49207139453e4953a409a65efe167400; no voters:
I20260808 06:57:09.107381 24960 raft_consensus.cc:2781] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:09.107429 24960 raft_consensus.cc:482] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:09.107456 24960 raft_consensus.cc:3037] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:09.108170 24960 raft_consensus.cc:504] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.108294 24960 leader_election.cc:288] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 election: Requested vote from peers 49207139453e4953a409a65efe167400 (127.21.28.65:37195), 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525)
I20260808 06:57:09.108441 24352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3cf29e3adb4c86a6176e3a1aeb5019" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49207139453e4953a409a65efe167400"
I20260808 06:57:09.108526 24352 raft_consensus.cc:3037] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:09.109326 24352 raft_consensus.cc:2454] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 1.
I20260808 06:57:09.109473 24550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3cf29e3adb4c86a6176e3a1aeb5019" candidate_uuid: "487586eb30df460e9d3416738c9f1314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a918e2e600d44268412dc6196c1f777"
I20260808 06:57:09.109514 24652 leader_election.cc:302] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487586eb30df460e9d3416738c9f1314, 49207139453e4953a409a65efe167400; no voters:
I20260808 06:57:09.109549 24550 raft_consensus.cc:3037] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:09.109627 24960 raft_consensus.cc:2781] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:09.109676 24960 raft_consensus.cc:686] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 1 LEADER]: Becoming Leader. State: Replica: 487586eb30df460e9d3416738c9f1314, State: Running, Role: LEADER
I20260808 06:57:09.109726 24960 consensus_queue.cc:237] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } }
I20260808 06:57:09.110249 24136 catalog_manager.cc:5495] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 reported cstate change: term changed from 0 to 1, leader changed from <none> to 487586eb30df460e9d3416738c9f1314 (127.21.28.67). New cstate: current_term: 1 leader_uuid: "487586eb30df460e9d3416738c9f1314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "487586eb30df460e9d3416738c9f1314" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42069 } health_report { overall_health: HEALTHY } } }
I20260808 06:57:09.110360 24550 raft_consensus.cc:2454] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 487586eb30df460e9d3416738c9f1314 in term 1.
I20260808 06:57:09.122100 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:09.119758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.21.28.67:42069)
0808 06:57:09.119803 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:09.119805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:09.119815 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:09.119861 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:09.119861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:09.119871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:09.119939 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:09.120027 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.120754 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.121423 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:09.121730 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:09.121805 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:09.121878 (+ 73us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:09.121920 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:09.121963 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:09.122005 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 06:57:09.122027 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:09.124249 24550 raft_consensus.cc:1261] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Refusing update from remote peer 487586eb30df460e9d3416738c9f1314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:09.124388 24352 raft_consensus.cc:1261] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 49207139453e4953a409a65efe167400 [term 1 FOLLOWER]: Refusing update from remote peer 487586eb30df460e9d3416738c9f1314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:09.124441 24960 consensus_queue.cc:1035] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a918e2e600d44268412dc6196c1f777" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:09.124553 24949 consensus_queue.cc:1035] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49207139453e4953a409a65efe167400" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 06:57:09.129168 24939 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 06:57:10.645814 24137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:10.645925 24137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:10.646697 24137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a24177cb2954ee18e2f8bfc35ee506e: Sending DeleteTablet for 3 replicas of tablet 6a3cf29e3adb4c86a6176e3a1aeb5019
I20260808 06:57:10.646951 24724 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:10 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:50200
I20260808 06:57:10.646943 24329 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:10 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:44742
I20260808 06:57:10.646983 24521 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:10 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:54550
I20260808 06:57:10.647096 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 24245
I20260808 06:57:10.654440 25064 tablet_replica.cc:330] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: stopping tablet replica
I20260808 06:57:10.654515 25064 raft_consensus.cc:2229] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:10.654556 25064 raft_consensus.cc:2258] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:10.654659 25065 tablet_replica.cc:330] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: stopping tablet replica
I20260808 06:57:10.654752 25065 raft_consensus.cc:2229] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:10.654825 25065 raft_consensus.cc:2258] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:10.654835 25064 ts_tablet_manager.cc:1905] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:10.655052 25065 ts_tablet_manager.cc:1905] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:10.656617 25065 ts_tablet_manager.cc:1918] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 06:57:10.656672 25065 log.cc:1196] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/6a3cf29e3adb4c86a6176e3a1aeb5019
I20260808 06:57:10.656685 25064 ts_tablet_manager.cc:1918] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 06:57:10.656772 25064 log.cc:1196] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/6a3cf29e3adb4c86a6176e3a1aeb5019
I20260808 06:57:10.656895 25065 ts_tablet_manager.cc:1939] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 487586eb30df460e9d3416738c9f1314: Deleting consensus metadata
I20260808 06:57:10.657003 25064 ts_tablet_manager.cc:1939] T 6a3cf29e3adb4c86a6176e3a1aeb5019 P 1a918e2e600d44268412dc6196c1f777: Deleting consensus metadata
I20260808 06:57:10.657223 24119 catalog_manager.cc:4841] TS 487586eb30df460e9d3416738c9f1314 (127.21.28.67:42069): tablet 6a3cf29e3adb4c86a6176e3a1aeb5019 (table test-table [id=74d6642ef3414302b1669bddc795e19e]) successfully deleted
I20260808 06:57:10.657477 24121 catalog_manager.cc:4841] TS 1a918e2e600d44268412dc6196c1f777 (127.21.28.66:45525): tablet 6a3cf29e3adb4c86a6176e3a1aeb5019 (table test-table [id=74d6642ef3414302b1669bddc795e19e]) successfully deleted
W20260808 06:57:10.660033 24119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:37195 (error 108)
I20260808 06:57:10.660308 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 24430
I20260808 06:57:10.667330 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:10.660384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:37195 (local address 127.0.0.1:44762)
0808 06:57:10.667243 (+ 6859us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:10.667293 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6832,"thread_start_us":61,"threads_started":1}
W20260808 06:57:10.667444 24119 catalog_manager.cc:4584] TS 49207139453e4953a409a65efe167400 (127.21.28.65:37195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a3cf29e3adb4c86a6176e3a1aeb5019: Network error: Client connection negotiation failed: client connection to 127.21.28.65:37195: connect: Connection refused (error 111)
I20260808 06:57:10.675633 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 24631
I20260808 06:57:10.683101 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 24099
2026-08-08T06:57:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5440 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:10 dist-test-slave-nj21 krb5kdc[25072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:10 dist-test-slave-nj21 krb5kdc[25072](info): set up 2 sockets
Aug 08 06:57:10 dist-test-slave-nj21 krb5kdc[25072](info): commencing operation
krb5kdc: starting...
W20260808 06:57:11.817945 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:11 dist-test-slave-nj21 krb5kdc[25072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:11.926306 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:45707
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:12.003974 25087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:12.004156 25087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:12.004176 25087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:12.005271 25087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:12.005312 25087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:12.005326 25087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:12.005349 25087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:12.005373 25087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:12.006770 25087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:12.006989 25087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:12.007184 25087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:12.011650 25098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:12.011662 25092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:12.011699 25095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:12.011752 25087 server_base.cc:1034] running on GCE node
I20260808 06:57:12.012095 25087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:12.012351 25087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:12.013481 25087 hybrid_clock.cc:648] HybridClock initialized: now 1786172232013474 us; error 34 us; skew 500 ppm
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172232, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:12.015730 25087 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:12.016137 25087 webserver.cc:460] Webserver started at http://127.21.28.126:32809/ using document root <none> and password file <none>
I20260808 06:57:12.016330 25087 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:12.016376 25087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:12.016508 25087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:12.017480 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "0cdd35a24d1442ada209d8fc483aba00"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "415887a31f8176e69a856e355c3255a2"
server_key_iv: "af8ace36034e35cfea9be9fe0dc68a47"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.017800 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "0cdd35a24d1442ada209d8fc483aba00"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "415887a31f8176e69a856e355c3255a2"
server_key_iv: "af8ace36034e35cfea9be9fe0dc68a47"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.018975 25087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 06:57:12.019625 25110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.019855 25087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:12.019937 25087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "0cdd35a24d1442ada209d8fc483aba00"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "415887a31f8176e69a856e355c3255a2"
server_key_iv: "af8ace36034e35cfea9be9fe0dc68a47"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.020004 25087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:12.027387 25087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:12.029083 25087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:12.029201 25087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:12.032848 25087 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:45707
I20260808 06:57:12.032899 25183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:45707 every 8 connection(s)
I20260808 06:57:12.033169 25087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:12.033969 25186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:12.036234 25186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00: Bootstrap starting.
I20260808 06:57:12.036794 25186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:12.037058 25186 log.cc:824] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:12.037750 25186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00: No bootstrap required, opened a new log
I20260808 06:57:12.039288 25186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdd35a24d1442ada209d8fc483aba00" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45707 } }
I20260808 06:57:12.039415 25186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:12.039445 25186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdd35a24d1442ada209d8fc483aba00, State: Initialized, Role: FOLLOWER
I20260808 06:57:12.039578 25186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdd35a24d1442ada209d8fc483aba00" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45707 } }
I20260808 06:57:12.039656 25186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:12.039732 25186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:12.039784 25186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:12.040037 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 25087
I20260808 06:57:12.040112 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:12.040256 21617 external_mini_cluster.cc:1428] Setting key 6b72ad8935ab5cccb0af441f76187f88
I20260808 06:57:12.040318 25186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdd35a24d1442ada209d8fc483aba00" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45707 } }
I20260808 06:57:12.040427 25186 leader_election.cc:302] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cdd35a24d1442ada209d8fc483aba00; no voters:
I20260808 06:57:12.040589 25186 leader_election.cc:288] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:12.040639 25193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:12.040812 25193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [term 1 LEADER]: Becoming Leader. State: Replica: 0cdd35a24d1442ada209d8fc483aba00, State: Running, Role: LEADER
I20260808 06:57:12.040900 25186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:12.040952 25193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdd35a24d1442ada209d8fc483aba00" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45707 } }
I20260808 06:57:12.041275 25193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cdd35a24d1442ada209d8fc483aba00. Latest consensus state: current_term: 1 leader_uuid: "0cdd35a24d1442ada209d8fc483aba00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdd35a24d1442ada209d8fc483aba00" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45707 } } }
I20260808 06:57:12.041325 25199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cdd35a24d1442ada209d8fc483aba00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cdd35a24d1442ada209d8fc483aba00" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45707 } } }
I20260808 06:57:12.041416 25193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:12.041424 25199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:12.041642 25204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:12.042333 25204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:12.044067 25204 catalog_manager.cc:1347] Generated new cluster ID: e2f1a5425d334a8f8450a9aeb59bf515
I20260808 06:57:12.044126 25204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:12.050644 25196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:12.040634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.21.28.126:45707)
0808 06:57:12.040817 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:12.040826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:12.040955 (+ 129us) server_negotiation.cc:407] Connection header received
0808 06:57:12.041158 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:12.041166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:12.041236 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:12.041347 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:12.042282 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.043794 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.044685 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.045117 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.045218 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:12.047510 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:12.047514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:12.047515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:12.047526 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:12.049159 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:12.049522 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:12.049525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:12.049525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:12.049635 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:12.049834 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:12.049834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:12.049835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:12.050064 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:12.050244 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:12.050313 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 06:57:12.050448 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260808 06:57:12.055836 25204 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:12.056329 25204 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:12.063902 25204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cdd35a24d1442ada209d8fc483aba00: Generated new TSK 0
I20260808 06:57:12.064059 25204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:12.096376 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45707
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:12.194041 25229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:12.194228 25229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:12.194250 25229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:12.195350 25229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:12.195389 25229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:12.195416 25229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:12.196820 25229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:12.197041 25229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:12.197249 25229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:12.201833 25236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:12.201858 25238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:12.201794 25235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:12.201972 25229 server_base.cc:1034] running on GCE node
I20260808 06:57:12.202145 25229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:12.202325 25229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:12.203452 25229 hybrid_clock.cc:648] HybridClock initialized: now 1786172232203447 us; error 33 us; skew 500 ppm
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172232, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:12.206005 25229 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:12.206362 25229 webserver.cc:460] Webserver started at http://127.21.28.65:37315/ using document root <none> and password file <none>
I20260808 06:57:12.206552 25229 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:12.206590 25229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:12.206676 25229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:12.207628 25229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "3461753a00cf41ddb02590f3f69d4c99"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "27f33af65c40a2390e0513e11c060819"
server_key_iv: "d1b3860fea70adb93727ab904894c0e7"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.207976 25229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "3461753a00cf41ddb02590f3f69d4c99"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "27f33af65c40a2390e0513e11c060819"
server_key_iv: "d1b3860fea70adb93727ab904894c0e7"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.209151 25229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:12.209786 25248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.209923 25229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:12.210005 25229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "3461753a00cf41ddb02590f3f69d4c99"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "27f33af65c40a2390e0513e11c060819"
server_key_iv: "d1b3860fea70adb93727ab904894c0e7"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.210062 25229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:12.218644 25229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:12.220434 25229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:12.220552 25229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:12.220736 25229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:12.221006 25229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:12.221038 25229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.221063 25229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:12.221087 25229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.226759 25229 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:40455
I20260808 06:57:12.226816 25397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:40455 every 8 connection(s)
I20260808 06:57:12.227074 25229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:12.231151 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 25229
I20260808 06:57:12.231241 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:12.231428 21617 external_mini_cluster.cc:1428] Setting key 0dd910dc766a8813242f39cb362c2233
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172232, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:12.235908 25196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:12.227661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:46369 (local address 127.21.28.126:45707)
0808 06:57:12.227776 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:12.227779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:12.227946 (+ 167us) server_negotiation.cc:407] Connection header received
0808 06:57:12.228419 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:12.228420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:12.228436 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:12.228483 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:12.229352 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.229861 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.230782 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.230988 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.231044 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:12.233167 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:12.233168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:12.233168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:12.233176 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:12.234563 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:12.235007 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:12.235008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:12.235008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:12.235060 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:12.235312 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:12.235312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:12.235313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:12.235410 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:12.235465 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:12.235770 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 06:57:12.235841 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 06:57:12.236126 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:12.227645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45707 (local address 127.21.28.65:46369)
0808 06:57:12.227830 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:12.227868 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:12.228322 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:12.228557 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:12.228561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:12.229094 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:12.229264 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:12.229268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.229942 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.229943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:12.230704 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:12.230706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.231056 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.231057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:12.231181 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:12.231665 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:12.231694 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:12.233085 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:12.234650 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:12.234657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:12.234659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:12.234940 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:12.235113 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:12.235114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:12.235114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:12.235259 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:12.235470 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:12.235476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:12.235624 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 06:57:12.235748 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 06:57:12.235909 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260808 06:57:12.236615 25398 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45707
I20260808 06:57:12.236708 25398 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:12.236889 25398 heartbeater.cc:511] Master 127.21.28.126:45707 requested a full tablet report, sending...
I20260808 06:57:12.237423 25133 ts_manager.cc:194] Registered new tserver with Master: 3461753a00cf41ddb02590f3f69d4c99 (127.21.28.65:40455)
I20260808 06:57:12.238294 25133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:46369
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:12.256138 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45707
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:12.334967 25405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:12.335153 25405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:12.335175 25405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:12.336336 25405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:12.336376 25405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:12.336401 25405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:12.337824 25405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:12.338027 25405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:12.338253 25405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:12.342918 25415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:12.342963 25410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:12.342948 25411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:12.343060 25405 server_base.cc:1034] running on GCE node
I20260808 06:57:12.343390 25405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:12.343605 25405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:12.344741 25405 hybrid_clock.cc:648] HybridClock initialized: now 1786172232344726 us; error 27 us; skew 500 ppm
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172232, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:12.347115 25405 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:12.347606 25405 webserver.cc:460] Webserver started at http://127.21.28.66:39797/ using document root <none> and password file <none>
I20260808 06:57:12.347821 25405 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:12.347867 25405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:12.347958 25405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:12.348874 25405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "fcbc4464976b4b5e9cadd12917323700"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "f18af69c70dba15daa5f266a301d561f"
server_key_iv: "157b9b8fab3b04507b8439756dc4aa42"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.349153 25405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "fcbc4464976b4b5e9cadd12917323700"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "f18af69c70dba15daa5f266a301d561f"
server_key_iv: "157b9b8fab3b04507b8439756dc4aa42"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.350332 25405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:12.350981 25429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.351117 25405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:12.351195 25405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "fcbc4464976b4b5e9cadd12917323700"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "f18af69c70dba15daa5f266a301d561f"
server_key_iv: "157b9b8fab3b04507b8439756dc4aa42"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.351265 25405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:12.365108 25405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:12.366670 25405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:12.366789 25405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:12.366979 25405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:12.367264 25405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:12.367297 25405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.367323 25405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:12.367348 25405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.373008 25405 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40527
I20260808 06:57:12.373050 25573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40527 every 8 connection(s)
I20260808 06:57:12.373335 25405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172232, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:12.380282 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 25405
I20260808 06:57:12.380371 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:12.380558 21617 external_mini_cluster.cc:1428] Setting key dba0dcb65af18b7780750c401a377c35
I20260808 06:57:12.382055 25196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:12.373951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:37805 (local address 127.21.28.126:45707)
0808 06:57:12.373997 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:12.373999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:12.374158 (+ 159us) server_negotiation.cc:407] Connection header received
0808 06:57:12.374660 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:12.374661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:12.374673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:12.374748 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:12.375624 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.376210 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.377139 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.377341 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.377404 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:12.379366 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:12.379367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:12.379367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:12.379375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:12.380632 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:12.381165 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:12.381167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:12.381167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:12.381230 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:12.381437 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:12.381438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:12.381438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:12.381553 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:12.381610 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:12.381863 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 06:57:12.381964 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:57:12.382184 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:12.373908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45707 (local address 127.21.28.66:37805)
0808 06:57:12.374052 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:12.374089 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:12.374541 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:12.374808 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:12.374812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:12.375357 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:12.375539 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:12.375544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.376288 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.376289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:12.377048 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:12.377050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.377407 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.377407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:12.377500 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:12.377967 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:12.377975 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:12.379276 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:12.380867 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:12.380873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:12.380875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:12.381094 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:12.381284 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:12.381285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:12.381285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:12.381381 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:12.381606 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:12.381612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:12.381744 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 06:57:12.381831 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 06:57:12.381982 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260808 06:57:12.382990 25576 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45707
I20260808 06:57:12.383102 25576 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:12.383266 25576 heartbeater.cc:511] Master 127.21.28.126:45707 requested a full tablet report, sending...
I20260808 06:57:12.383608 25133 ts_manager.cc:194] Registered new tserver with Master: fcbc4464976b4b5e9cadd12917323700 (127.21.28.66:40527)
I20260808 06:57:12.383985 25133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:37805
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:12.404466 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45707
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:12.482788 25581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:12.482970 25581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:12.482991 25581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:12.484119 25581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:12.484157 25581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:12.484182 25581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:12.485532 25581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:12.485702 25581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:12.485862 25581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:12.490392 25588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:12.490443 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:12.490459 25589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:12.490725 25581 server_base.cc:1034] running on GCE node
I20260808 06:57:12.490916 25581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:12.491117 25581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:12.492238 25581 hybrid_clock.cc:648] HybridClock initialized: now 1786172232492226 us; error 31 us; skew 500 ppm
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172232, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:12.494606 25581 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:12.494999 25581 webserver.cc:460] Webserver started at http://127.21.28.67:37963/ using document root <none> and password file <none>
I20260808 06:57:12.495187 25581 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:12.495230 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:12.495340 25581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:12.496295 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "be5b4f89f08347a6bc19633d17cbf508"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "24e63e288d0d9d657baec61a9a2ff6d4"
server_key_iv: "df162fa8035f2f5482638529af62977f"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.496599 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "be5b4f89f08347a6bc19633d17cbf508"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "24e63e288d0d9d657baec61a9a2ff6d4"
server_key_iv: "df162fa8035f2f5482638529af62977f"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.497853 25581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:12.498539 25603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.498661 25581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:12.498735 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "be5b4f89f08347a6bc19633d17cbf508"
format_stamp: "Formatted at 2026-08-08 06:57:12 on dist-test-slave-nj21"
server_key: "24e63e288d0d9d657baec61a9a2ff6d4"
server_key_iv: "df162fa8035f2f5482638529af62977f"
server_key_version: "encryptionkey@0"
I20260808 06:57:12.498795 25581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:12.512208 25581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:12.514135 25581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:12.514257 25581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:12.514453 25581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:12.514725 25581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:12.514758 25581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.514784 25581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:12.514807 25581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:12.520465 25581 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:36929
I20260808 06:57:12.520503 25760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:36929 every 8 connection(s)
I20260808 06:57:12.520767 25581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172232, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:12.528508 25196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:12.521374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:48181 (local address 127.21.28.126:45707)
0808 06:57:12.521422 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:12.521425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:12.521591 (+ 166us) server_negotiation.cc:407] Connection header received
0808 06:57:12.522021 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:12.522021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:12.522030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:12.522080 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:12.522885 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.523392 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.524302 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.524491 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.524543 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:12.526346 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:12.526348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:12.526348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:12.526356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:12.527386 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:12.527748 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:12.527750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:12.527750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:12.527810 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:12.528003 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:12.528003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:12.528003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:12.528102 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:12.528148 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:12.528375 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 06:57:12.528444 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 06:57:12.528584 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:12.521337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45707 (local address 127.21.28.67:48181)
0808 06:57:12.521478 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:12.521516 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:12.521945 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:12.522134 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:12.522136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:12.522644 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:12.522813 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:12.522818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.523451 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.523451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:12.524227 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:12.524229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:12.524547 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:12.524548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:12.524635 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:12.525032 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:12.525040 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:12.526263 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:12.527444 (+ 1181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:12.527449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:12.527450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:12.527636 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:12.527866 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:12.527866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:12.527866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:12.527958 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:12.528143 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:12.528145 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:12.528243 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 06:57:12.528317 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 06:57:12.528431 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260808 06:57:12.528923 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 25581
I20260808 06:57:12.529021 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:12.529119 25761 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45707
I20260808 06:57:12.529204 25761 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:12.529220 21617 external_mini_cluster.cc:1428] Setting key 0ecc1402a727b74f5184ec30b005dcfe
I20260808 06:57:12.529371 25761 heartbeater.cc:511] Master 127.21.28.126:45707 requested a full tablet report, sending...
I20260808 06:57:12.529716 25133 ts_manager.cc:194] Registered new tserver with Master: be5b4f89f08347a6bc19633d17cbf508 (127.21.28.67:36929)
I20260808 06:57:12.530110 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:57:12.530153 25133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:48181
W20260808 06:57:12.542137 25196 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:12.541202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.21.28.126:45707)
0808 06:57:12.541265 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:12.541267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:12.541364 (+ 97us) server_negotiation.cc:407] Connection header received
0808 06:57:12.541407 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:12.541407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:12.541416 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:12.541494 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:12.542072 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60798: BlockingRecv error: recv got EOF from 127.0.0.1:60798 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 06:57:12.542994 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 25229
I20260808 06:57:12.573619 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 25405
I20260808 06:57:12.581189 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 25581
I20260808 06:57:12.593321 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 25087
2026-08-08T06:57:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1910 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25778](info): set up 2 sockets
Aug 08 06:57:12 dist-test-slave-nj21 krb5kdc[25778](info): commencing operation
krb5kdc: starting...
W20260808 06:57:13.721073 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:13 dist-test-slave-nj21 krb5kdc[25778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:13.830343 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:34335
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:13.908767 25796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:13.908981 25796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:13.909014 25796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:13.910154 25796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:13.910194 25796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:13.910213 25796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:13.910229 25796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:13.910244 25796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:13.911700 25796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:34335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:13.911945 25796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:13.912138 25796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:13.916587 25808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:13.916663 25796 server_base.cc:1034] running on GCE node
W20260808 06:57:13.916605 25806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:13.916574 25810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:13.916970 25796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:13.917263 25796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:13.918396 25796 hybrid_clock.cc:648] HybridClock initialized: now 1786172233918381 us; error 33 us; skew 500 ppm
Aug 08 06:57:13 dist-test-slave-nj21 krb5kdc[25778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172233, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:13.920719 25796 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:13.921074 25796 webserver.cc:460] Webserver started at http://127.21.28.126:45785/ using document root <none> and password file <none>
I20260808 06:57:13.921234 25796 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:13.921269 25796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:13.921355 25796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:13.922277 25796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "df2e051fdafc45dc92919ee3e814b07d"
format_stamp: "Formatted at 2026-08-08 06:57:13 on dist-test-slave-nj21"
server_key: "1410f96f122d3ce52ff8e42499ea0b77"
server_key_iv: "c270070fdab95b7e63bbb84d9abe4b34"
server_key_version: "encryptionkey@0"
I20260808 06:57:13.922564 25796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "df2e051fdafc45dc92919ee3e814b07d"
format_stamp: "Formatted at 2026-08-08 06:57:13 on dist-test-slave-nj21"
server_key: "1410f96f122d3ce52ff8e42499ea0b77"
server_key_iv: "c270070fdab95b7e63bbb84d9abe4b34"
server_key_version: "encryptionkey@0"
I20260808 06:57:13.923704 25796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:13.924337 25821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:13.924495 25796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:13.924579 25796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "df2e051fdafc45dc92919ee3e814b07d"
format_stamp: "Formatted at 2026-08-08 06:57:13 on dist-test-slave-nj21"
server_key: "1410f96f122d3ce52ff8e42499ea0b77"
server_key_iv: "c270070fdab95b7e63bbb84d9abe4b34"
server_key_version: "encryptionkey@0"
I20260808 06:57:13.924629 25796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:13.942914 25796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:13.944576 25796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:13.944676 25796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:13.948201 25796 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:34335
I20260808 06:57:13.948237 25894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:34335 every 8 connection(s)
I20260808 06:57:13.948712 25796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:13.949283 25895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:13.951418 25895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d: Bootstrap starting.
I20260808 06:57:13.951962 25895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:13.952195 25895 log.cc:824] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:13.952754 25895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d: No bootstrap required, opened a new log
I20260808 06:57:13.954224 25895 raft_consensus.cc:348] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2e051fdafc45dc92919ee3e814b07d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34335 } }
I20260808 06:57:13.954393 25895 raft_consensus.cc:374] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:13.954397 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 25796
I20260808 06:57:13.954437 25895 raft_consensus.cc:729] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2e051fdafc45dc92919ee3e814b07d, State: Initialized, Role: FOLLOWER
I20260808 06:57:13.954484 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:13.954577 25895 consensus_queue.cc:260] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2e051fdafc45dc92919ee3e814b07d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34335 } }
I20260808 06:57:13.954651 25895 raft_consensus.cc:388] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:13.954667 21617 external_mini_cluster.cc:1428] Setting key 3e3ad345380716cf05d2ce0eb3c0215d
I20260808 06:57:13.954686 25895 raft_consensus.cc:482] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:13.954734 25895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:13.955331 25895 raft_consensus.cc:504] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2e051fdafc45dc92919ee3e814b07d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34335 } }
I20260808 06:57:13.955420 25895 leader_election.cc:302] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df2e051fdafc45dc92919ee3e814b07d; no voters:
I20260808 06:57:13.955571 25895 leader_election.cc:288] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:13.955610 25903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:13.955775 25903 raft_consensus.cc:686] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [term 1 LEADER]: Becoming Leader. State: Replica: df2e051fdafc45dc92919ee3e814b07d, State: Running, Role: LEADER
I20260808 06:57:13.955864 25895 sys_catalog.cc:564] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:13.955919 25903 consensus_queue.cc:237] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2e051fdafc45dc92919ee3e814b07d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34335 } }
I20260808 06:57:13.956245 25903 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df2e051fdafc45dc92919ee3e814b07d. Latest consensus state: current_term: 1 leader_uuid: "df2e051fdafc45dc92919ee3e814b07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2e051fdafc45dc92919ee3e814b07d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34335 } } }
I20260808 06:57:13.956318 25903 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:13.956467 25906 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df2e051fdafc45dc92919ee3e814b07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2e051fdafc45dc92919ee3e814b07d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34335 } } }
I20260808 06:57:13.956526 25906 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d [sys.catalog]: This master's current role is: LEADER
W20260808 06:57:13.957523 25926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:57:13.957576 25926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:57:13.958238 25912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:13.958392 25912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:13.959870 25912 catalog_manager.cc:1347] Generated new cluster ID: 702f9c9e565c4393a8e11d1c9af0cbb3
I20260808 06:57:13.959913 25912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:57:13 dist-test-slave-nj21 krb5kdc[25778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:13.966324 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:13.955440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.21.28.126:34335)
0808 06:57:13.955611 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:13.955620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:13.955653 (+ 33us) server_negotiation.cc:407] Connection header received
0808 06:57:13.955826 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:13.955833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:13.955887 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:13.956821 (+ 934us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:13.956932 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:13.958107 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:13.959118 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:13.959709 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:13.959790 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:13.962303 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:13.962307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:13.962307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:13.962318 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:13.964803 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:13.965160 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:13.965163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:13.965163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:13.965265 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:13.965429 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:13.965429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:13.965429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:13.965718 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:13.965863 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:13.965987 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 06:57:13.966120 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260808 06:57:13.967351 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:57:13.980509 25912 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:13.981045 25912 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:13.984000 25912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df2e051fdafc45dc92919ee3e814b07d: Generated new TSK 0
I20260808 06:57:13.984167 25912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:57:13.984949 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:13.980957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.21.28.126:34335)
0808 06:57:13.980993 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:13.980995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:13.981147 (+ 152us) server_negotiation.cc:407] Connection header received
0808 06:57:13.981238 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:13.981239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:13.981248 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:13.981308 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:13.981709 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:13.982448 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:13.983496 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:13.983811 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:13.983875 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:13.984567 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:13.984568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:13.984569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:13.984727 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:13.984741 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:13.984747 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:13.984792 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:13.984852 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 06:57:13.984873 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:57:13.986946 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 25796
2026-08-08T06:57:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1391 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:14 dist-test-slave-nj21 krb5kdc[25945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:14 dist-test-slave-nj21 krb5kdc[25945](info): set up 2 sockets
Aug 08 06:57:14 dist-test-slave-nj21 krb5kdc[25945](info): commencing operation
krb5kdc: starting...
W20260808 06:57:15.108575 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:15.216555 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:39009
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:39009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:15.293861 25962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:15.294056 25962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:15.294087 25962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:15.295173 25962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:15.295210 25962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:15.295229 25962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:15.295245 25962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:15.295260 25962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:15.296705 25962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:39009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:15.296934 25962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:15.297122 25962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:15.301496 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:15.301525 25962 server_base.cc:1034] running on GCE node
W20260808 06:57:15.301523 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:15.301496 25970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:15.301868 25962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:15.302143 25962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:15.303282 25962 hybrid_clock.cc:648] HybridClock initialized: now 1786172235303269 us; error 30 us; skew 500 ppm
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:15.305516 25962 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:15.305912 25962 webserver.cc:460] Webserver started at http://127.21.28.126:32891/ using document root <none> and password file <none>
I20260808 06:57:15.306116 25962 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:15.306160 25962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:15.306277 25962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:15.307201 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "829f59a5fb564995acb47be9d24f5843"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "34f8945f3d229691dd3ef7317ee2a583"
server_key_iv: "9c73042eee40e2f79d7006f95529d3bf"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.307520 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "829f59a5fb564995acb47be9d24f5843"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "34f8945f3d229691dd3ef7317ee2a583"
server_key_iv: "9c73042eee40e2f79d7006f95529d3bf"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.308756 25962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:15.309376 25981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.309532 25962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:15.309602 25962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "829f59a5fb564995acb47be9d24f5843"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "34f8945f3d229691dd3ef7317ee2a583"
server_key_iv: "9c73042eee40e2f79d7006f95529d3bf"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.309666 25962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:15.328689 25962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:15.330256 25962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:15.330384 25962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:15.334080 25962 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:39009
I20260808 06:57:15.334115 26067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:39009 every 8 connection(s)
I20260808 06:57:15.334424 25962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:15.335505 26068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:15.337721 26068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843: Bootstrap starting.
I20260808 06:57:15.338279 26068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:15.338521 26068 log.cc:824] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:15.339129 26068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843: No bootstrap required, opened a new log
I20260808 06:57:15.340596 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 25962
I20260808 06:57:15.340703 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:15.340894 21617 external_mini_cluster.cc:1428] Setting key 1ed2be751708bcbbf714dd1b54c88fa9
I20260808 06:57:15.340891 26068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829f59a5fb564995acb47be9d24f5843" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 39009 } }
I20260808 06:57:15.341055 26068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:15.341099 26068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829f59a5fb564995acb47be9d24f5843, State: Initialized, Role: FOLLOWER
I20260808 06:57:15.341238 26068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829f59a5fb564995acb47be9d24f5843" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 39009 } }
I20260808 06:57:15.341326 26068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:15.341421 26068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:15.341488 26068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:15.342402 26068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829f59a5fb564995acb47be9d24f5843" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 39009 } }
I20260808 06:57:15.342502 26068 leader_election.cc:302] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 829f59a5fb564995acb47be9d24f5843; no voters:
I20260808 06:57:15.342661 26068 leader_election.cc:288] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:15.342696 26075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:15.342862 26075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [term 1 LEADER]: Becoming Leader. State: Replica: 829f59a5fb564995acb47be9d24f5843, State: Running, Role: LEADER
I20260808 06:57:15.342900 26068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:15.343017 26075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829f59a5fb564995acb47be9d24f5843" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 39009 } }
I20260808 06:57:15.343370 26079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "829f59a5fb564995acb47be9d24f5843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829f59a5fb564995acb47be9d24f5843" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 39009 } } }
I20260808 06:57:15.343448 26079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:15.344066 26086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:15.344393 26076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 829f59a5fb564995acb47be9d24f5843. Latest consensus state: current_term: 1 leader_uuid: "829f59a5fb564995acb47be9d24f5843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "829f59a5fb564995acb47be9d24f5843" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 39009 } } }
I20260808 06:57:15.344471 26076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:15.344543 26086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:15.346244 26086 catalog_manager.cc:1347] Generated new cluster ID: 6acd62cda2cd40149c6b31e68634497e
I20260808 06:57:15.346294 26086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:15.351423 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:15.341383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.21.28.126:39009)
0808 06:57:15.341584 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:15.341593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:15.341626 (+ 33us) server_negotiation.cc:407] Connection header received
0808 06:57:15.341732 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:15.341737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:15.341775 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:15.341883 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:15.342467 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.343777 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.344811 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.345314 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.345419 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:15.348094 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:15.348098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:15.348099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:15.348110 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:15.349906 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.350319 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.350322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.350322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.350424 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.350607 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.350607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.350608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.350866 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:15.351012 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:15.351115 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 06:57:15.351235 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260808 06:57:15.355656 26086 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:15.356132 26086 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:15.366909 26086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 829f59a5fb564995acb47be9d24f5843: Generated new TSK 0
I20260808 06:57:15.367058 26086 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:15.396118 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:39009
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:15.474705 26106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:15.474893 26106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:15.474915 26106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:15.475989 26106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:15.476022 26106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:15.476044 26106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:15.477424 26106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:39009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:15.477598 26106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:15.477763 26106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:15.482445 26119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:15.482504 26116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:15.482546 26106 server_base.cc:1034] running on GCE node
W20260808 06:57:15.482468 26115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:15.482802 26106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:15.483014 26106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:15.484141 26106 hybrid_clock.cc:648] HybridClock initialized: now 1786172235484126 us; error 32 us; skew 500 ppm
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:15.486472 26106 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:15.486874 26106 webserver.cc:460] Webserver started at http://127.21.28.65:42691/ using document root <none> and password file <none>
I20260808 06:57:15.487074 26106 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:15.487118 26106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:15.487237 26106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:15.488272 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "cc4eb0143888430fafaee18e7d8f8eb4"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "7c9ac9039531e975697ce616b6ab0f82"
server_key_iv: "0cd5dd55a32201559a72897b73f3420e"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.488601 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "cc4eb0143888430fafaee18e7d8f8eb4"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "7c9ac9039531e975697ce616b6ab0f82"
server_key_iv: "0cd5dd55a32201559a72897b73f3420e"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.489854 26106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:15.490504 26128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.490646 26106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:15.490722 26106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "cc4eb0143888430fafaee18e7d8f8eb4"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "7c9ac9039531e975697ce616b6ab0f82"
server_key_iv: "0cd5dd55a32201559a72897b73f3420e"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.490790 26106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:15.507607 26106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:15.509292 26106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:15.509418 26106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:15.509634 26106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:15.509944 26106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:15.509976 26106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.510004 26106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:15.510027 26106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.515864 26106 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:35479
I20260808 06:57:15.515934 26304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:35479 every 8 connection(s)
I20260808 06:57:15.516202 26106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:15.520475 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 26106
I20260808 06:57:15.520569 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:15.520784 21617 external_mini_cluster.cc:1428] Setting key 56b0e329bf1bc35f4356cc3c9c8125a8
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:15.524808 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:15.516831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:43201 (local address 127.21.28.126:39009)
0808 06:57:15.516888 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:15.516890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:15.517101 (+ 211us) server_negotiation.cc:407] Connection header received
0808 06:57:15.517563 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:15.517564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:15.517578 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:15.517631 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:15.518501 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.519005 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.519978 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.520170 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.520218 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:15.522267 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:15.522269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:15.522269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:15.522277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:15.523499 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.523945 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.523947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.523947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.524006 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.524250 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.524251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.524251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.524348 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:15.524394 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:15.524680 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 06:57:15.524747 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 06:57:15.525027 26307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:15.516798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:39009 (local address 127.21.28.65:43201)
0808 06:57:15.516971 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:15.517012 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:15.517488 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:15.517684 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:15.517687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:15.518245 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:15.518432 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:15.518437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.519065 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.519066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:15.519913 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:15.519916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.520225 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.520226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:15.520338 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:15.520835 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:15.520846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:15.522181 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:15.523574 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:15.523581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:15.523583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:15.523867 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:15.524065 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:15.524066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:15.524066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:15.524195 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:15.524399 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:15.524404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:15.524558 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 06:57:15.524658 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 06:57:15.524804 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260808 06:57:15.525502 26305 heartbeater.cc:347] Connected to a master server at 127.21.28.126:39009
I20260808 06:57:15.525586 26305 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:15.525748 26305 heartbeater.cc:511] Master 127.21.28.126:39009 requested a full tablet report, sending...
I20260808 06:57:15.526243 26006 ts_manager.cc:194] Registered new tserver with Master: cc4eb0143888430fafaee18e7d8f8eb4 (127.21.28.65:35479)
I20260808 06:57:15.526823 26006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:43201
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:15.544479 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:39009
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:15.626057 26310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:15.626248 26310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:15.626271 26310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:15.627326 26310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:15.627357 26310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:15.627380 26310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:15.628751 26310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:39009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:15.628928 26310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:15.629103 26310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:15.633788 26322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:15.633845 26320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:15.633999 26310 server_base.cc:1034] running on GCE node
W20260808 06:57:15.633846 26319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:15.634234 26310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:15.634434 26310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:15.635552 26310 hybrid_clock.cc:648] HybridClock initialized: now 1786172235635544 us; error 26 us; skew 500 ppm
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:15.637952 26310 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:15.638360 26310 webserver.cc:460] Webserver started at http://127.21.28.66:43101/ using document root <none> and password file <none>
I20260808 06:57:15.638553 26310 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:15.638602 26310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:15.638724 26310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:15.639722 26310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "9c4284a8b0e6461ba8ca8f04c533595c"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "521d4a7dc4e1f01651d37117f3f7b501"
server_key_iv: "1e7775f1ad4d6be064f4961f596291be"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.640057 26310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "9c4284a8b0e6461ba8ca8f04c533595c"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "521d4a7dc4e1f01651d37117f3f7b501"
server_key_iv: "1e7775f1ad4d6be064f4961f596291be"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.641258 26310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:15.641877 26333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.642020 26310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:15.642104 26310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "9c4284a8b0e6461ba8ca8f04c533595c"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "521d4a7dc4e1f01651d37117f3f7b501"
server_key_iv: "1e7775f1ad4d6be064f4961f596291be"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.642167 26310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:15.670359 26310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:15.672415 26310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:15.672545 26310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:15.672750 26310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:15.673038 26310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:15.673070 26310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.673098 26310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:15.673121 26310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.679965 26310 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:38793
I20260808 06:57:15.680325 26310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:15.680826 26504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:38793 every 8 connection(s)
I20260808 06:57:15.684341 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 26310
I20260808 06:57:15.684446 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:15.684646 21617 external_mini_cluster.cc:1428] Setting key 78376057eecbda3c7bf95b3dd9dd9f2b
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:15.694598 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:15.684831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:48729 (local address 127.21.28.126:39009)
0808 06:57:15.684878 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:15.684880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:15.684891 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:57:15.684917 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:15.684918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:15.684930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:15.684993 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:15.686166 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.686931 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.688323 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.688606 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.688656 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:15.691356 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:15.691358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:15.691358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:15.691367 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:15.693056 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.693566 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.693568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.693568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.693643 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.693900 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.693901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.693902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.694051 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:15.694133 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:15.694382 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 06:57:15.694499 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:15.694654 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:15.683980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:39009 (local address 127.21.28.66:48729)
0808 06:57:15.684150 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:15.684195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:15.684772 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:15.685077 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:15.685082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:15.685773 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:15.686048 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:15.686055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.687042 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.687043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:15.688220 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:15.688224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.688750 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.688751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:15.688883 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:15.689379 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:15.689390 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:15.691253 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:15.693157 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:15.693164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:15.693166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:15.693493 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:15.693728 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:15.693729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:15.693729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:15.693834 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:15.694118 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:15.694120 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:15.694225 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 06:57:15.694308 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:57:15.694470 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260808 06:57:15.695098 26505 heartbeater.cc:347] Connected to a master server at 127.21.28.126:39009
I20260808 06:57:15.695219 26505 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:15.695416 26505 heartbeater.cc:511] Master 127.21.28.126:39009 requested a full tablet report, sending...
I20260808 06:57:15.695852 26006 ts_manager.cc:194] Registered new tserver with Master: 9c4284a8b0e6461ba8ca8f04c533595c (127.21.28.66:38793)
I20260808 06:57:15.696213 26006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:48729
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:15.714138 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:39009
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:15.793598 26512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:15.793792 26512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:15.793823 26512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:15.794951 26512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:15.794991 26512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:15.795025 26512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:15.796458 26512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:39009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:15.796692 26512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:15.796885 26512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:15.801532 26519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:15.801582 26522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:15.801584 26512 server_base.cc:1034] running on GCE node
W20260808 06:57:15.801533 26520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:15.801929 26512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:15.802134 26512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:15.803268 26512 hybrid_clock.cc:648] HybridClock initialized: now 1786172235803256 us; error 30 us; skew 500 ppm
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:15.809108 26512 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:15.809478 26512 webserver.cc:460] Webserver started at http://127.21.28.67:45019/ using document root <none> and password file <none>
I20260808 06:57:15.809681 26512 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:15.809723 26512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:15.809814 26512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:15.810753 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "bcfb8400c21f4f06997ff9fa24815f16"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "a76809432ff9f797fc7ddba367487fe3"
server_key_iv: "32cb8a72cce0cc85b2afa298968dbc25"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.811136 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "bcfb8400c21f4f06997ff9fa24815f16"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "a76809432ff9f797fc7ddba367487fe3"
server_key_iv: "32cb8a72cce0cc85b2afa298968dbc25"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.812502 26512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:15.813155 26530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.813300 26512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:15.813380 26512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "bcfb8400c21f4f06997ff9fa24815f16"
format_stamp: "Formatted at 2026-08-08 06:57:15 on dist-test-slave-nj21"
server_key: "a76809432ff9f797fc7ddba367487fe3"
server_key_iv: "32cb8a72cce0cc85b2afa298968dbc25"
server_key_version: "encryptionkey@0"
I20260808 06:57:15.813462 26512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:15.831521 26512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:15.833109 26512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:15.833225 26512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:15.833418 26512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:15.833684 26512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:15.833715 26512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.833737 26512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:15.833760 26512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:15.839336 26512 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:42805
I20260808 06:57:15.839403 26687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:42805 every 8 connection(s)
I20260808 06:57:15.839645 26512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:15.847954 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:15.840247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:56247 (local address 127.21.28.126:39009)
0808 06:57:15.840292 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:15.840294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:15.840491 (+ 197us) server_negotiation.cc:407] Connection header received
0808 06:57:15.840987 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:15.840989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:15.841002 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:15.841053 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:15.841937 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.842431 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.843407 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.843605 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.843659 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:15.845691 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:15.845693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:15.845693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:15.845700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:15.846735 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.847076 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.847077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.847078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.847132 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.847334 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.847335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.847335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.847450 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:15.847507 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:15.847775 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 06:57:15.847854 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:15.848098 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:15.840226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:39009 (local address 127.21.28.67:56247)
0808 06:57:15.840371 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:15.840404 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:15.840890 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:15.841121 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:15.841125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:15.841667 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:15.841837 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:15.841841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.842512 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.842513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:15.843325 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:15.843327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.843689 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.843691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:15.843800 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:15.844224 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:15.844234 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:15.845609 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:15.846804 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:15.846811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:15.846813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:15.847012 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:15.847185 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:15.847185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:15.847186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:15.847279 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:15.847504 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:15.847510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:15.847634 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 06:57:15.847750 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 06:57:15.847897 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260808 06:57:15.848554 26688 heartbeater.cc:347] Connected to a master server at 127.21.28.126:39009
I20260808 06:57:15.848644 26688 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:15.848655 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 26512
I20260808 06:57:15.848755 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:15.848822 26688 heartbeater.cc:511] Master 127.21.28.126:39009 requested a full tablet report, sending...
I20260808 06:57:15.848933 21617 external_mini_cluster.cc:1428] Setting key 8d42236905d3ddbdd657f1894d6255c9
I20260808 06:57:15.849265 26006 ts_manager.cc:194] Registered new tserver with Master: bcfb8400c21f4f06997ff9fa24815f16 (127.21.28.67:42805)
I20260808 06:57:15.849633 26006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:56247
I20260808 06:57:15.849776 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[25945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172235, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:15.871033 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:15.864818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.21.28.126:39009)
0808 06:57:15.864863 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:15.864865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:15.864986 (+ 121us) server_negotiation.cc:407] Connection header received
0808 06:57:15.865054 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:15.865054 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:15.865066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:15.865108 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:15.865626 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.866100 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.866806 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:15.866997 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:15.867053 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:15.869020 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:15.869022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:15.869022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:15.869029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:15.870040 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.870318 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.870319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.870320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.870368 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:15.870532 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:15.870533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:15.870533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:15.870668 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:15.870755 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:15.870861 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 06:57:15.870947 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 06:57:15.871232 26006 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33340
I20260808 06:57:15.871583 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 26106
I20260808 06:57:15.883060 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 26310
I20260808 06:57:15.889261 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 26512
I20260808 06:57:15.897631 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 25962
2026-08-08T06:57:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1909 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[26704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[26704](info): set up 2 sockets
Aug 08 06:57:15 dist-test-slave-nj21 krb5kdc[26704](info): commencing operation
krb5kdc: starting...
W20260808 06:57:17.013482 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:17.122073 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:42597
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:42597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:17.200184 26724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:17.200371 26724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:17.200389 26724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:17.201464 26724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:17.201496 26724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:17.201509 26724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:17.201520 26724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:17.201534 26724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:17.202944 26724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:42597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:42597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:17.203123 26724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:17.203293 26724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:57:17.207883 26724 server_base.cc:1034] running on GCE node
W20260808 06:57:17.207873 26732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:17.207856 26733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:17.207864 26735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:17.208276 26724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:17.208544 26724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:17.209695 26724 hybrid_clock.cc:648] HybridClock initialized: now 1786172237209680 us; error 33 us; skew 500 ppm
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:17.212013 26724 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:17.212426 26724 webserver.cc:460] Webserver started at http://127.21.28.126:33685/ using document root <none> and password file <none>
I20260808 06:57:17.212630 26724 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:17.212677 26724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:17.212797 26724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:17.213750 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "8b163133043b42da99066ddeb85ab4fd"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "db0ba6684ee6916a6019156d1ad6273b"
server_key_iv: "c6d67926e876845525b965a95b234429"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.214069 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "8b163133043b42da99066ddeb85ab4fd"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "db0ba6684ee6916a6019156d1ad6273b"
server_key_iv: "c6d67926e876845525b965a95b234429"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.215230 26724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 06:57:17.215864 26746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.216007 26724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.216089 26724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "8b163133043b42da99066ddeb85ab4fd"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "db0ba6684ee6916a6019156d1ad6273b"
server_key_iv: "c6d67926e876845525b965a95b234429"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.216143 26724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:17.228493 26724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:17.230048 26724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:17.230170 26724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:17.233862 26724 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:42597
I20260808 06:57:17.233896 26812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:42597 every 8 connection(s)
I20260808 06:57:17.234354 26724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:17.234951 26813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:17.235713 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 26724
I20260808 06:57:17.235821 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:17.236023 21617 external_mini_cluster.cc:1428] Setting key f1218c4264ccbb404a333f4730fc0d11
I20260808 06:57:17.237742 26813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd: Bootstrap starting.
I20260808 06:57:17.238286 26813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:17.238526 26813 log.cc:824] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:17.239115 26813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd: No bootstrap required, opened a new log
I20260808 06:57:17.240986 26813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b163133043b42da99066ddeb85ab4fd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42597 } }
I20260808 06:57:17.241112 26813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:17.241146 26813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b163133043b42da99066ddeb85ab4fd, State: Initialized, Role: FOLLOWER
I20260808 06:57:17.241269 26813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b163133043b42da99066ddeb85ab4fd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42597 } }
I20260808 06:57:17.241343 26813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:17.241382 26813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:17.241427 26813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 0 FOLLOWER]: Advancing to term 1
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:17.241943 26813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b163133043b42da99066ddeb85ab4fd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42597 } }
I20260808 06:57:17.242035 26813 leader_election.cc:302] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b163133043b42da99066ddeb85ab4fd; no voters:
I20260808 06:57:17.242183 26813 leader_election.cc:288] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:17.242224 26819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:17.242372 26819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [term 1 LEADER]: Becoming Leader. State: Replica: 8b163133043b42da99066ddeb85ab4fd, State: Running, Role: LEADER
I20260808 06:57:17.242439 26813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:17.242465 26819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b163133043b42da99066ddeb85ab4fd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42597 } }
I20260808 06:57:17.242761 26819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b163133043b42da99066ddeb85ab4fd. Latest consensus state: current_term: 1 leader_uuid: "8b163133043b42da99066ddeb85ab4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b163133043b42da99066ddeb85ab4fd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42597 } } }
I20260808 06:57:17.242836 26819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:17.243017 26822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b163133043b42da99066ddeb85ab4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b163133043b42da99066ddeb85ab4fd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42597 } } }
I20260808 06:57:17.243125 26822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:17.243057 26830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:17.243710 26830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:17.245981 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.236493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.21.28.126:42597)
0808 06:57:17.236661 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:17.236668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:17.236691 (+ 23us) server_negotiation.cc:407] Connection header received
0808 06:57:17.237081 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:17.237086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:17.237127 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:17.237240 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:17.237825 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.238775 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.239516 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.239974 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.240067 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.242189 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:17.242193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:17.242194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:17.242204 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:17.244421 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.244771 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.244774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.244775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.244878 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.245100 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.245100 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.245101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.245387 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:17.245530 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:17.245634 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 06:57:17.245786 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260808 06:57:17.246212 26830 catalog_manager.cc:1347] Generated new cluster ID: 1d66ccc3cf494fd2b301bd0c01a283ae
I20260808 06:57:17.246255 26830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:17.264883 26830 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:17.265377 26830 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:17.267562 26830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b163133043b42da99066ddeb85ab4fd: Generated new TSK 0
I20260808 06:57:17.267737 26830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:17.293939 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42597
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:17.372337 26851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:17.372519 26851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:17.372540 26851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:17.373636 26851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:17.373675 26851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:17.373700 26851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:17.375124 26851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:17.375321 26851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:17.375523 26851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:17.380092 26861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:17.380087 26864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:17.380215 26851 server_base.cc:1034] running on GCE node
W20260808 06:57:17.380106 26860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:17.380481 26851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:17.380697 26851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:17.381830 26851 hybrid_clock.cc:648] HybridClock initialized: now 1786172237381816 us; error 32 us; skew 500 ppm
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:17.384236 26851 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:17.384747 26851 webserver.cc:460] Webserver started at http://127.21.28.65:43147/ using document root <none> and password file <none>
I20260808 06:57:17.384934 26851 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:17.384975 26851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:17.385089 26851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:17.386044 26851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "6199ac7f7d11412e950bc7f62fc00f9f"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "ae910f91770dfd209947992a2c2aaaa2"
server_key_iv: "a1d4be37fe1a70b72a8a4933304fc32d"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.386356 26851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "6199ac7f7d11412e950bc7f62fc00f9f"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "ae910f91770dfd209947992a2c2aaaa2"
server_key_iv: "a1d4be37fe1a70b72a8a4933304fc32d"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.387537 26851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:17.388211 26874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.388350 26851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.388424 26851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "6199ac7f7d11412e950bc7f62fc00f9f"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "ae910f91770dfd209947992a2c2aaaa2"
server_key_iv: "a1d4be37fe1a70b72a8a4933304fc32d"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.388486 26851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:17.404083 26851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:17.405645 26851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:17.405761 26851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:17.405970 26851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:17.406243 26851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:17.406275 26851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.406296 26851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:17.406318 26851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.412042 26851 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:40777
I20260808 06:57:17.412086 27023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:40777 every 8 connection(s)
I20260808 06:57:17.412369 26851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:17.418169 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 26851
I20260808 06:57:17.418272 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:17.418473 21617 external_mini_cluster.cc:1428] Setting key 84bb25bb5d27d70ab36db30006008088
I20260808 06:57:17.420745 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.413014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:38245 (local address 127.21.28.126:42597)
0808 06:57:17.413069 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:17.413071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:17.413215 (+ 144us) server_negotiation.cc:407] Connection header received
0808 06:57:17.414053 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:17.414054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:17.414063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:17.414114 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:17.414627 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.415123 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.416041 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.416234 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.416277 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.418028 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:17.418029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:17.418029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:17.418035 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:17.419452 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.419844 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.419845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.419846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.419925 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.420145 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.420145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.420146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.420273 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:17.420331 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:17.420569 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 06:57:17.420656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 06:57:17.420900 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.412968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42597 (local address 127.21.28.65:38245)
0808 06:57:17.413109 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:17.413146 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:17.413976 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:17.414164 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:17.414167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:17.414398 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:17.414570 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:17.414573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.415180 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.415180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:17.415978 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:17.415981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.416289 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.416290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:17.416379 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.416735 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:17.416742 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:17.417947 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:17.419523 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:17.419529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:17.419531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:17.419786 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:17.419987 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:17.419988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:17.419988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:17.420095 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:17.420330 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:17.420333 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:17.420459 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 06:57:17.420542 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:57:17.420695 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260808 06:57:17.421355 27025 heartbeater.cc:347] Connected to a master server at 127.21.28.126:42597
I20260808 06:57:17.421450 27025 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:17.421617 27025 heartbeater.cc:511] Master 127.21.28.126:42597 requested a full tablet report, sending...
I20260808 06:57:17.422060 26766 ts_manager.cc:194] Registered new tserver with Master: 6199ac7f7d11412e950bc7f62fc00f9f (127.21.28.65:40777)
I20260808 06:57:17.422640 26766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:38245
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:17.442445 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42597
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:17.520148 27033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:17.520334 27033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:17.520355 27033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:17.521474 27033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:17.521513 27033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:17.521541 27033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:17.522976 27033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:17.523180 27033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:17.523375 27033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:17.528045 27040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:17.528090 27041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:17.528126 27033 server_base.cc:1034] running on GCE node
W20260808 06:57:17.528097 27044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:17.528441 27033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:17.528656 27033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:17.529789 27033 hybrid_clock.cc:648] HybridClock initialized: now 1786172237529777 us; error 26 us; skew 500 ppm
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:17.532107 27033 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:17.532485 27033 webserver.cc:460] Webserver started at http://127.21.28.66:33381/ using document root <none> and password file <none>
I20260808 06:57:17.532667 27033 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:17.532711 27033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:17.532832 27033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:17.533789 27033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "4e82a8e277fd45b7aae5d7f91eb139a6"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "eb21ec7f77c14f7753cfc4681c3d6d2d"
server_key_iv: "01c660dfc2e16489a7f5d292e95b2fee"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.534116 27033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "4e82a8e277fd45b7aae5d7f91eb139a6"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "eb21ec7f77c14f7753cfc4681c3d6d2d"
server_key_iv: "01c660dfc2e16489a7f5d292e95b2fee"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.535285 27033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:17.535926 27054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.536084 27033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:17.536170 27033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "4e82a8e277fd45b7aae5d7f91eb139a6"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "eb21ec7f77c14f7753cfc4681c3d6d2d"
server_key_iv: "01c660dfc2e16489a7f5d292e95b2fee"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.536237 27033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:17.548494 27033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:17.550076 27033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:17.550190 27033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:17.550392 27033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:17.550669 27033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:17.550700 27033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.550721 27033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:17.550745 27033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.556320 27033 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:34843
I20260808 06:57:17.556381 27220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:34843 every 8 connection(s)
I20260808 06:57:17.556643 27033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:17.564272 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.557275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:47169 (local address 127.21.28.126:42597)
0808 06:57:17.557346 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:17.557348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:17.557489 (+ 141us) server_negotiation.cc:407] Connection header received
0808 06:57:17.558324 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:17.558325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:17.558334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:17.558377 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:17.558896 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.559366 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.560262 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.560453 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.560496 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.562265 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:17.562267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:17.562267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:17.562273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:17.563291 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.563600 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.563601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.563601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.563643 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.563827 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.563827 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.563827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.563921 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:17.563967 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:17.564153 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 06:57:17.564210 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:57:17.564414 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.557239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42597 (local address 127.21.28.66:47169)
0808 06:57:17.557384 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:17.557419 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:17.558249 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:17.558436 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:17.558439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:17.558658 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:17.558839 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:17.558843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.559424 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.559425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:17.560196 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:17.560198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.560504 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.560504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:17.560595 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.560967 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:17.560974 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:17.562190 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:17.563350 (+ 1160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:17.563356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:17.563357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:17.563548 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:17.563702 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:17.563703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:17.563703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:17.563785 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:17.563962 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:17.563965 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:17.564064 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 06:57:17.564136 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 06:57:17.564242 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260808 06:57:17.564808 27221 heartbeater.cc:347] Connected to a master server at 127.21.28.126:42597
I20260808 06:57:17.564888 27221 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:17.565032 27221 heartbeater.cc:511] Master 127.21.28.126:42597 requested a full tablet report, sending...
I20260808 06:57:17.565356 26766 ts_manager.cc:194] Registered new tserver with Master: 4e82a8e277fd45b7aae5d7f91eb139a6 (127.21.28.66:34843)
I20260808 06:57:17.565690 26766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:47169
I20260808 06:57:17.566447 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 27033
I20260808 06:57:17.566535 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:17.566674 21617 external_mini_cluster.cc:1428] Setting key c10bc6555deb655d79e5ee4236174707
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:17.590094 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42597
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:17.668824 27228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:17.669003 27228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:17.669034 27228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:17.670146 27228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:17.670185 27228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:17.670218 27228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:17.671656 27228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:17.671885 27228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:17.672089 27228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:17.677008 27234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:17.677115 27238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:17.677008 27235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:17.677426 27228 server_base.cc:1034] running on GCE node
I20260808 06:57:17.677603 27228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:17.677824 27228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:17.681032 27228 hybrid_clock.cc:648] HybridClock initialized: now 1786172237681005 us; error 36 us; skew 500 ppm
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:17.683326 27228 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:17.683773 27228 webserver.cc:460] Webserver started at http://127.21.28.67:43639/ using document root <none> and password file <none>
I20260808 06:57:17.683995 27228 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:17.684041 27228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:17.684163 27228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:17.685111 27228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "8a8664727cda4f8a941aae348584cf2f"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "5be70175af97d4ef8ed314f7de8a1372"
server_key_iv: "5cfe1d81cec2de780aba4df5ef3ad528"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.685425 27228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "8a8664727cda4f8a941aae348584cf2f"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "5be70175af97d4ef8ed314f7de8a1372"
server_key_iv: "5cfe1d81cec2de780aba4df5ef3ad528"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.686610 27228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:17.687224 27245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.687384 27228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:17.687466 27228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "8a8664727cda4f8a941aae348584cf2f"
format_stamp: "Formatted at 2026-08-08 06:57:17 on dist-test-slave-nj21"
server_key: "5be70175af97d4ef8ed314f7de8a1372"
server_key_iv: "5cfe1d81cec2de780aba4df5ef3ad528"
server_key_version: "encryptionkey@0"
I20260808 06:57:17.687529 27228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:17.713629 27228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:17.715277 27228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:17.715410 27228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:17.715617 27228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:17.715932 27228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:17.715967 27228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.715994 27228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:17.716012 27228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:17.721650 27228 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:39365
I20260808 06:57:17.721691 27409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:39365 every 8 connection(s)
I20260808 06:57:17.721990 27228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:17.724261 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 27228
I20260808 06:57:17.724359 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:17.724546 21617 external_mini_cluster.cc:1428] Setting key 71cd2b5f85bdfec5a4f93eddf4a03958
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:17.730319 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.722615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:55739 (local address 127.21.28.126:42597)
0808 06:57:17.722687 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:17.722689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:17.722846 (+ 157us) server_negotiation.cc:407] Connection header received
0808 06:57:17.723745 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:17.723746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:17.723757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:17.723801 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:17.724598 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.725179 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.726136 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.726337 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.726391 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.728296 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:17.728297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:17.728297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:17.728303 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:17.729340 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.729669 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.729669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.729670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.729714 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.729878 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.729878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.729878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.729970 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:17.730014 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:17.730200 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 06:57:17.730256 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 06:57:17.730479 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.722590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42597 (local address 127.21.28.67:55739)
0808 06:57:17.722741 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:17.722773 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:17.723626 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:17.723889 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:17.723892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:17.724231 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:17.724513 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:17.724520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.725265 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.725266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:17.726057 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:17.726060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.726411 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.726412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:17.726540 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.726908 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:17.726915 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:17.728219 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:17.729400 (+ 1181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:17.729406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:17.729407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:17.729616 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:17.729755 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:17.729756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:17.729756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:17.729839 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:17.730008 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:17.730011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:17.730111 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 06:57:17.730183 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 06:57:17.730290 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260808 06:57:17.730898 27410 heartbeater.cc:347] Connected to a master server at 127.21.28.126:42597
I20260808 06:57:17.730988 27410 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:17.731153 27410 heartbeater.cc:511] Master 127.21.28.126:42597 requested a full tablet report, sending...
I20260808 06:57:17.731477 26766 ts_manager.cc:194] Registered new tserver with Master: 8a8664727cda4f8a941aae348584cf2f (127.21.28.67:39365)
I20260808 06:57:17.731878 26766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:55739
I20260808 06:57:17.735944 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:17.758327 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.751372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.21.28.65:40777)
0808 06:57:17.751530 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:17.751535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:17.751544 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:17.751568 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:17.751570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:17.751582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:17.751654 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:17.752211 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.752961 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.753675 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.753989 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.754046 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.756075 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:17.756084 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:17.756084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:17.756092 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:17.757277 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.757588 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.757591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.757591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.757643 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.757800 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.757801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.757801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.757936 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:17.758064 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:17.758141 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 06:57:17.758216 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:17.765208 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:17.759566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.21.28.126:42597)
0808 06:57:17.759617 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:17.759619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:17.759627 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:17.759715 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:17.759716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:17.759726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:17.759771 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:17.760178 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.760643 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.761253 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:17.761444 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:17.761495 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:17.763354 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:17.763355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:17.763355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:17.763362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:17.764389 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.764653 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.764654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.764654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.764700 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:17.764824 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:17.764824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:17.764824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:17.764955 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:17.765022 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:17.765088 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 06:57:17.765146 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260808 06:57:17.765407 26766 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37986
I20260808 06:57:17.765699 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 26851
I20260808 06:57:17.782506 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 27033
I20260808 06:57:17.795769 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 27228
I20260808 06:57:17.802539 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 26724
2026-08-08T06:57:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1904 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[27429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[27429](info): set up 2 sockets
Aug 08 06:57:17 dist-test-slave-nj21 krb5kdc[27429](info): commencing operation
krb5kdc: starting...
W20260808 06:57:18.932891 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:19.043529 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46827
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:19.121934 27448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:19.122135 27448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:19.122167 27448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:19.123317 27448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:19.123355 27448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:19.123373 27448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:19.123389 27448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:19.123404 27448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:19.124876 27448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:19.125125 27448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:19.125322 27448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:19.129849 27457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:19.129817 27456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:19.129837 27459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:19.129908 27448 server_base.cc:1034] running on GCE node
I20260808 06:57:19.130239 27448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:19.130522 27448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:19.131695 27448 hybrid_clock.cc:648] HybridClock initialized: now 1786172239131666 us; error 46 us; skew 500 ppm
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:19.134171 27448 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:19.134521 27448 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:19.134573 27448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:19.134696 27448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:19.135727 27448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "c36e246d4d994fab9ef20b1480c61980"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "86c468246cbe58fc72c76d9ca47a748b"
server_key_iv: "92dd799cd8a4d1a95bb77c062b423322"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.136065 27448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "c36e246d4d994fab9ef20b1480c61980"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "86c468246cbe58fc72c76d9ca47a748b"
server_key_iv: "92dd799cd8a4d1a95bb77c062b423322"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.137241 27448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:19.137914 27466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.138062 27448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:19.138147 27448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "c36e246d4d994fab9ef20b1480c61980"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "86c468246cbe58fc72c76d9ca47a748b"
server_key_iv: "92dd799cd8a4d1a95bb77c062b423322"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.138221 27448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:19.169903 27448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:19.172137 27448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:19.172281 27448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:19.176008 27448 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46827
I20260808 06:57:19.176038 27537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46827 every 8 connection(s)
I20260808 06:57:19.176311 27448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:19.177366 27538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:19.177510 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 27448
I20260808 06:57:19.177610 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:19.177819 21617 external_mini_cluster.cc:1428] Setting key acee420e469472d658ed47b68e505ea1
I20260808 06:57:19.180604 27538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980: Bootstrap starting.
I20260808 06:57:19.181274 27538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:19.181566 27538 log.cc:824] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:19.182268 27538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980: No bootstrap required, opened a new log
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:19.184252 27538 raft_consensus.cc:348] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e246d4d994fab9ef20b1480c61980" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46827 } }
I20260808 06:57:19.184403 27538 raft_consensus.cc:374] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:19.184443 27538 raft_consensus.cc:729] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36e246d4d994fab9ef20b1480c61980, State: Initialized, Role: FOLLOWER
I20260808 06:57:19.184576 27538 consensus_queue.cc:260] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e246d4d994fab9ef20b1480c61980" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46827 } }
I20260808 06:57:19.184648 27538 raft_consensus.cc:388] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:19.184680 27538 raft_consensus.cc:482] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:19.184726 27538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:19.185276 27538 raft_consensus.cc:504] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e246d4d994fab9ef20b1480c61980" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46827 } }
I20260808 06:57:19.185384 27538 leader_election.cc:302] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c36e246d4d994fab9ef20b1480c61980; no voters:
I20260808 06:57:19.185539 27538 leader_election.cc:288] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:19.185567 27546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:19.185707 27546 raft_consensus.cc:686] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [term 1 LEADER]: Becoming Leader. State: Replica: c36e246d4d994fab9ef20b1480c61980, State: Running, Role: LEADER
I20260808 06:57:19.185786 27546 consensus_queue.cc:237] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e246d4d994fab9ef20b1480c61980" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46827 } }
I20260808 06:57:19.185791 27538 sys_catalog.cc:564] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:19.186218 27549 sys_catalog.cc:455] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c36e246d4d994fab9ef20b1480c61980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e246d4d994fab9ef20b1480c61980" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46827 } } }
I20260808 06:57:19.186224 27550 sys_catalog.cc:455] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c36e246d4d994fab9ef20b1480c61980. Latest consensus state: current_term: 1 leader_uuid: "c36e246d4d994fab9ef20b1480c61980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36e246d4d994fab9ef20b1480c61980" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46827 } } }
I20260808 06:57:19.186311 27549 sys_catalog.cc:458] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:19.186318 27550 sys_catalog.cc:458] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:19.186681 27554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:19.187367 27554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:19.188956 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.178356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.21.28.126:46827)
0808 06:57:19.178520 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.178529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.178556 (+ 27us) server_negotiation.cc:407] Connection header received
0808 06:57:19.179021 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.179027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.179075 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.179189 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:19.179950 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.180895 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.181893 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.182365 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.182494 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.184851 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:19.184855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:19.184855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:19.184867 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:19.187127 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.187531 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.187534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.187535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.187646 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.187857 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.187858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.187858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.188144 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:19.188270 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.188636 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.188766 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260808 06:57:19.188971 27554 catalog_manager.cc:1347] Generated new cluster ID: 92d96289c0ea49c3907f4cd06687fa3e
I20260808 06:57:19.189023 27554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:19.199379 27554 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:19.199937 27554 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:19.202549 27554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980: Generated new TSK 0
I20260808 06:57:19.202692 27554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:19.232188 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46827
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:19.312254 27574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:19.312448 27574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:19.312481 27574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:19.313599 27574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:19.313637 27574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:19.313670 27574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:19.315114 27574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:19.315335 27574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:19.315539 27574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:19.320174 27581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:19.320204 27587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:19.320216 27574 server_base.cc:1034] running on GCE node
W20260808 06:57:19.320317 27583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:19.320564 27574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:19.320770 27574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:19.321925 27574 hybrid_clock.cc:648] HybridClock initialized: now 1786172239321909 us; error 29 us; skew 500 ppm
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:19.324402 27574 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:19.324767 27574 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:19.324823 27574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:19.324947 27574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:19.325932 27574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "6560fe41305c4f72bff33aa2eeda1940"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "9e1981dc12e85a756858965ae19e9a42"
server_key_iv: "e6ee8af8de974be75b00015cad7307bc"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.326243 27574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "6560fe41305c4f72bff33aa2eeda1940"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "9e1981dc12e85a756858965ae19e9a42"
server_key_iv: "e6ee8af8de974be75b00015cad7307bc"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.327410 27574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:19.328130 27596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.328291 27574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.328379 27574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "6560fe41305c4f72bff33aa2eeda1940"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "9e1981dc12e85a756858965ae19e9a42"
server_key_iv: "e6ee8af8de974be75b00015cad7307bc"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.328440 27574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:19.336500 27574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:19.338440 27574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:19.338562 27574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:19.338752 27574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:19.339025 27574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:19.339056 27574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.339082 27574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:19.339107 27574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.344841 27574 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:40861
I20260808 06:57:19.344899 27757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:40861 every 8 connection(s)
I20260808 06:57:19.345121 27574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:19.346900 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 27574
I20260808 06:57:19.347018 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:19.347215 21617 external_mini_cluster.cc:1428] Setting key b433abf638c2705f4272bc70cbb4b068
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:19.354349 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.345668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:51365 (local address 127.21.28.126:46827)
0808 06:57:19.345709 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.345711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.345900 (+ 189us) server_negotiation.cc:407] Connection header received
0808 06:57:19.346416 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.346417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.346430 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.346482 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:19.347583 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.348179 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.349500 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.349794 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.349855 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.351901 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:19.351902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:19.351903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:19.351909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:19.353074 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.353521 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.353522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.353522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.353567 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.353809 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.353809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.353809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.353899 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:19.353943 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.354227 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.354289 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 06:57:19.354604 27761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.345660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46827 (local address 127.21.28.65:51365)
0808 06:57:19.345794 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.345826 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:19.346317 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:19.346546 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:19.346550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:19.346828 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:19.347484 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.347490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.348265 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.348266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.349404 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.349407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.349870 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.349870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.349975 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.350369 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:19.350376 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:19.351819 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:19.353149 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:19.353157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:19.353159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:19.353449 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:19.353617 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:19.353618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:19.353618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:19.353753 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:19.353950 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:19.353955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:19.354108 (+ 153us) client_negotiation.cc:769] Sending connection context
0808 06:57:19.354217 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 06:57:19.354373 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260808 06:57:19.355073 27759 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46827
I20260808 06:57:19.355158 27759 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:19.355290 27759 heartbeater.cc:511] Master 127.21.28.126:46827 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
I20260808 06:57:19.355844 27489 ts_manager.cc:194] Registered new tserver with Master: 6560fe41305c4f72bff33aa2eeda1940 (127.21.28.65:40861)
I20260808 06:57:19.356493 27489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:51365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:19.370249 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46827
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:19.447745 27765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:19.447933 27765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:19.447956 27765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:19.449054 27765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:19.449090 27765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:19.449118 27765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:19.450568 27765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:19.450773 27765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:19.450987 27765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:19.455644 27774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:19.455701 27775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:19.455631 27777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:19.455875 27765 server_base.cc:1034] running on GCE node
I20260808 06:57:19.456104 27765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:19.456318 27765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:19.457450 27765 hybrid_clock.cc:648] HybridClock initialized: now 1786172239457437 us; error 27 us; skew 500 ppm
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:19.459774 27765 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:19.460078 27765 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:19.460126 27765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:19.460249 27765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:19.461195 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "829644394b004a4289df42dafc4dcdae"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "800d71f285e43c38281d1c9e07bdd64a"
server_key_iv: "62216165de5ca6e1ff44ba082fd412f6"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.461510 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "829644394b004a4289df42dafc4dcdae"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "800d71f285e43c38281d1c9e07bdd64a"
server_key_iv: "62216165de5ca6e1ff44ba082fd412f6"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.462641 27765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 06:57:19.463235 27787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.463382 27765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.463464 27765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "829644394b004a4289df42dafc4dcdae"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "800d71f285e43c38281d1c9e07bdd64a"
server_key_iv: "62216165de5ca6e1ff44ba082fd412f6"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.463527 27765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:19.492879 27765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:19.494503 27765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:19.494633 27765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:19.494838 27765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:19.495137 27765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:19.495177 27765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.495204 27765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:19.495221 27765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.500735 27765 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:39871
I20260808 06:57:19.500789 27956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:39871 every 8 connection(s)
I20260808 06:57:19.501025 27765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:19.504350 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 27765
I20260808 06:57:19.504453 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:19.504626 21617 external_mini_cluster.cc:1428] Setting key aa275bd8afce1612023736b42d97fc60
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:19.509706 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.501631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:36445 (local address 127.21.28.126:46827)
0808 06:57:19.501682 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.501684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.501857 (+ 173us) server_negotiation.cc:407] Connection header received
0808 06:57:19.502342 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.502343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.502357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.502421 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:19.503311 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.503851 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.504861 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.505060 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.505116 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.507192 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:19.507194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:19.507194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:19.507200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:19.508390 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.508802 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.508803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.508803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.508848 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.509096 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.509096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.509097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.509222 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:19.509273 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.509574 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.509643 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 06:57:19.509946 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.501609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46827 (local address 127.21.28.66:36445)
0808 06:57:19.501743 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.501777 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:19.502245 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:19.502491 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:19.502495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:19.502740 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:19.503214 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.503219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.503932 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.503933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.504783 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.504785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.505126 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.505128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.505253 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.505683 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:19.505691 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:19.507108 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:19.508472 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:19.508479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:19.508481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:19.508736 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:19.508906 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:19.508907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:19.508908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:19.509036 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:19.509279 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:19.509285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:19.509439 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 06:57:19.509562 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 06:57:19.509721 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260808 06:57:19.510412 27957 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46827
I20260808 06:57:19.510505 27957 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:19.510658 27957 heartbeater.cc:511] Master 127.21.28.126:46827 requested a full tablet report, sending...
I20260808 06:57:19.511003 27489 ts_manager.cc:194] Registered new tserver with Master: 829644394b004a4289df42dafc4dcdae (127.21.28.66:39871)
I20260808 06:57:19.511358 27489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:36445
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:19.527750 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46827
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:19.607508 27966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:19.607712 27966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:19.607736 27966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:19.608850 27966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:19.608887 27966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:19.608916 27966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:19.610387 27966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:19.610592 27966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:19.610780 27966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:19.615350 27973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:19.615352 27977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:19.615370 27971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:19.615463 27966 server_base.cc:1034] running on GCE node
I20260808 06:57:19.615774 27966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:19.615972 27966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:19.617095 27966 hybrid_clock.cc:648] HybridClock initialized: now 1786172239617087 us; error 25 us; skew 500 ppm
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:19.619414 27966 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:19.619701 27966 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:19.619758 27966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:19.619849 27966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:19.620767 27966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "d93b0b49a6674be493397aabb106ff0d"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "347f9717fa0f1e4fdceeacd5f7a04c5e"
server_key_iv: "254bc052c9249ce3aa8687befdd01a5b"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.621042 27966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "d93b0b49a6674be493397aabb106ff0d"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "347f9717fa0f1e4fdceeacd5f7a04c5e"
server_key_iv: "254bc052c9249ce3aa8687befdd01a5b"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.622140 27966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:19.622758 27987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.622892 27966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:19.622973 27966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "d93b0b49a6674be493397aabb106ff0d"
format_stamp: "Formatted at 2026-08-08 06:57:19 on dist-test-slave-nj21"
server_key: "347f9717fa0f1e4fdceeacd5f7a04c5e"
server_key_iv: "254bc052c9249ce3aa8687befdd01a5b"
server_key_version: "encryptionkey@0"
I20260808 06:57:19.623034 27966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:19.632120 27966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:19.633972 27966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:19.634091 27966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:19.634281 27966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:19.634553 27966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:19.634584 27966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.634603 27966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:19.634626 27966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:19.640177 27966 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:40257
I20260808 06:57:19.640210 28152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:40257 every 8 connection(s)
I20260808 06:57:19.640450 27966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:19.641687 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 27966
I20260808 06:57:19.641798 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:19.642005 21617 external_mini_cluster.cc:1428] Setting key 1e55bd3dd0253465f6c486ffdd8a6674
Aug 08 06:57:19 dist-test-slave-nj21 krb5kdc[27429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:19.648703 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.641098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:58335 (local address 127.21.28.126:46827)
0808 06:57:19.641147 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.641150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.641342 (+ 192us) server_negotiation.cc:407] Connection header received
0808 06:57:19.641946 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.641947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.641957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.642005 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:19.643049 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.643534 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.644475 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.644682 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.644732 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.646489 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:19.646490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:19.646490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:19.646497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:19.647756 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.648066 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.648066 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.648067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.648110 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.648271 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.648271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.648272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.648361 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:19.648411 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.648590 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.648644 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 06:57:19.648856 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.641097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46827 (local address 127.21.28.67:58335)
0808 06:57:19.641233 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.641264 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:19.641836 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:19.642083 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:19.642086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:19.642405 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:19.642981 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.642986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.643593 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.643593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.644407 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.644410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.644738 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.644739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.644830 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.645175 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:19.645182 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:19.646406 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:19.647809 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:19.647813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:19.647814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:19.648012 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:19.648151 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:19.648152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:19.648152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:19.648232 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:19.648400 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:19.648402 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:19.648500 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 06:57:19.648571 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 06:57:19.648673 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260808 06:57:19.649243 28153 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46827
I20260808 06:57:19.649322 28153 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:19.649466 28153 heartbeater.cc:511] Master 127.21.28.126:46827 requested a full tablet report, sending...
I20260808 06:57:19.649799 27489 ts_manager.cc:194] Registered new tserver with Master: d93b0b49a6674be493397aabb106ff0d (127.21.28.67:40257)
I20260808 06:57:19.650153 27489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:58335
I20260808 06:57:19.653426 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:57:19.659601 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.654594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.21.28.126:46827)
0808 06:57:19.654645 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.654646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.654768 (+ 122us) server_negotiation.cc:407] Connection header received
0808 06:57:19.654823 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.654824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.654833 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.654873 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:19.655389 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.655879 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.656551 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.656738 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.656789 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.657724 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:19.657725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:19.657725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:19.657731 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:19.658639 (+ 908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.658924 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.658925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.658926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.658972 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:19.659127 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:19.659127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:19.659128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:19.659255 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:19.659338 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.659456 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.659523 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:57:19.660746 27489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:19.662197 27489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:19.668025 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.664049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40861 (local address 127.0.0.1:49658)
0808 06:57:19.664462 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.664470 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:19.664531 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:19.664687 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:19.664689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:19.664692 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:19.664748 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.664749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.666125 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.666126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.667044 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.667046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.667756 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.667757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.667834 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.667845 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 06:57:19.667905 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 06:57:19.667954 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":4,"thread_start_us":49,"threads_started":1}
I20260808 06:57:19.668111 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.664050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:39871 (local address 127.0.0.1:53690)
0808 06:57:19.664403 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.664441 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:19.664506 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:19.664772 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:19.664773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:19.664775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:19.664828 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.664829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.665762 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.665763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.666960 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.666963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.667874 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.667875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.667947 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.667951 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:19.668002 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 06:57:19.668041 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"spinlock_wait_cycles":1408,"thread_start_us":90,"threads_started":1}
I20260808 06:57:19.668045 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.664239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.21.28.65:40861)
0808 06:57:19.664393 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.664397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.664534 (+ 137us) server_negotiation.cc:407] Connection header received
0808 06:57:19.664595 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.664597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.664611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.664677 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:19.664860 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.666025 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.667118 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.667666 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.667756 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.667815 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.667929 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.667968 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260808 06:57:19.668309 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.664054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:40257 (local address 127.0.0.1:51074)
0808 06:57:19.664609 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.664617 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:19.664665 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:19.664979 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:19.664979 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:19.664981 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:19.665013 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.665014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.666186 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.666187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.667235 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.667237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.668112 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.668112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.668166 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.668169 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:19.668219 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 06:57:19.668254 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":62,"threads_started":1}
I20260808 06:57:19.668394 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.664252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.21.28.66:39871)
0808 06:57:19.664423 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.664426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.664510 (+ 84us) server_negotiation.cc:407] Connection header received
0808 06:57:19.664588 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.664590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.664607 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.664680 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:19.664908 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.665681 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.667028 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.667722 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.667786 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.667840 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.668277 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.668318 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260808 06:57:19.668556 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.664231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.21.28.67:40257)
0808 06:57:19.664369 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.664374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.664716 (+ 342us) server_negotiation.cc:407] Connection header received
0808 06:57:19.664815 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.664817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.664830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.664894 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:19.665070 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.666099 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.667308 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.667984 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.668036 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.668082 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.668447 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.668486 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260808 06:57:19.669026 27860 tablet_service.cc:1467] Processing CreateTablet for tablet 2855916982c34ef8b0e670a8704ae8ec (DEFAULT_TABLE table=test-table [id=d436d2ff9f5f4ee68345d99a1896cdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:19.669000 27667 tablet_service.cc:1467] Processing CreateTablet for tablet 2855916982c34ef8b0e670a8704ae8ec (DEFAULT_TABLE table=test-table [id=d436d2ff9f5f4ee68345d99a1896cdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:19.669116 28059 tablet_service.cc:1467] Processing CreateTablet for tablet 2855916982c34ef8b0e670a8704ae8ec (DEFAULT_TABLE table=test-table [id=d436d2ff9f5f4ee68345d99a1896cdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:19.669345 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2855916982c34ef8b0e670a8704ae8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:19.669345 27667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2855916982c34ef8b0e670a8704ae8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:19.669413 28059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2855916982c34ef8b0e670a8704ae8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:19.671756 28173 tablet_bootstrap.cc:492] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Bootstrap starting.
I20260808 06:57:19.672366 28173 tablet_bootstrap.cc:654] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:19.672576 28174 tablet_bootstrap.cc:492] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Bootstrap starting.
I20260808 06:57:19.673095 28173 log.cc:824] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:19.673199 28174 tablet_bootstrap.cc:654] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:19.673713 28174 log.cc:824] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:19.673844 28173 tablet_bootstrap.cc:492] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: No bootstrap required, opened a new log
I20260808 06:57:19.673913 28173 ts_tablet_manager.cc:1397] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:19.674054 28175 tablet_bootstrap.cc:492] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940: Bootstrap starting.
I20260808 06:57:19.674351 28174 tablet_bootstrap.cc:492] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: No bootstrap required, opened a new log
I20260808 06:57:19.674420 28174 ts_tablet_manager.cc:1397] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:19.674957 28175 tablet_bootstrap.cc:654] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:19.675520 28175 log.cc:824] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:19.675863 28173 raft_consensus.cc:348] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.675976 28173 raft_consensus.cc:374] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:19.675993 28173 raft_consensus.cc:729] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93b0b49a6674be493397aabb106ff0d, State: Initialized, Role: FOLLOWER
I20260808 06:57:19.676101 28173 consensus_queue.cc:260] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.676363 28153 heartbeater.cc:503] Master 127.21.28.126:46827 was elected leader, sending a full tablet report...
I20260808 06:57:19.676352 28173 ts_tablet_manager.cc:1428] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 06:57:19.676419 28174 raft_consensus.cc:348] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.676573 28174 raft_consensus.cc:374] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:19.676604 28174 raft_consensus.cc:729] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829644394b004a4289df42dafc4dcdae, State: Initialized, Role: FOLLOWER
I20260808 06:57:19.676712 28174 consensus_queue.cc:260] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.676970 28174 ts_tablet_manager.cc:1428] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 06:57:19.677568 28175 tablet_bootstrap.cc:492] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940: No bootstrap required, opened a new log
I20260808 06:57:19.677611 28175 ts_tablet_manager.cc:1397] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260808 06:57:19.678572 27957 heartbeater.cc:503] Master 127.21.28.126:46827 was elected leader, sending a full tablet report...
I20260808 06:57:19.679220 28175 raft_consensus.cc:348] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.679368 28175 raft_consensus.cc:374] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:19.679399 28175 raft_consensus.cc:729] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6560fe41305c4f72bff33aa2eeda1940, State: Initialized, Role: FOLLOWER
I20260808 06:57:19.679512 28175 consensus_queue.cc:260] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.679814 27759 heartbeater.cc:503] Master 127.21.28.126:46827 was elected leader, sending a full tablet report...
I20260808 06:57:19.679812 28175 ts_tablet_manager.cc:1428] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260808 06:57:19.751526 27959 tablet.cc:2369] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:57:19.845721 27760 tablet.cc:2369] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:19.859145 28181 raft_consensus.cc:482] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:19.859247 28181 raft_consensus.cc:504] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.859594 28181 leader_election.cc:288] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 829644394b004a4289df42dafc4dcdae (127.21.28.66:39871), d93b0b49a6674be493397aabb106ff0d (127.21.28.67:40257)
I20260808 06:57:19.862352 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.859860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:59305 (local address 127.21.28.66:39871)
0808 06:57:19.859934 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.859937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.860013 (+ 76us) server_negotiation.cc:407] Connection header received
0808 06:57:19.860083 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.860084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.860093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.860146 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:19.860286 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.860730 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.861671 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.862054 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.862116 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.862124 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.862249 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.862288 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 06:57:19.862353 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.859873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:53573 (local address 127.21.28.67:40257)
0808 06:57:19.859936 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.859938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.859949 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:57:19.860023 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.860024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.860033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.860087 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:19.860226 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.860703 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.861678 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.862050 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.862109 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.862119 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.862252 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.862287 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 06:57:19.862365 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.859762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:39871 (local address 127.21.28.65:59305)
0808 06:57:19.859941 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.859951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:19.860020 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:19.860154 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:19.860155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:19.860156 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:19.860223 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.860224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.860801 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.860802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.861587 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.861589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.862123 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.862124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.862189 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.862192 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:19.862236 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 06:57:19.862273 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":4,"thread_start_us":53,"threads_started":1}
I20260808 06:57:19.862377 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.859762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:40257 (local address 127.21.28.65:53573)
0808 06:57:19.859888 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.859900 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:19.859957 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:19.860095 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:19.860096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:19.860098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:19.860163 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.860165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.860777 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.860778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.861592 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:19.861594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.862124 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.862125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:19.862191 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.862194 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:19.862240 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 06:57:19.862280 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":22,"thread_start_us":57,"threads_started":1}
I20260808 06:57:19.862668 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2855916982c34ef8b0e670a8704ae8ec" candidate_uuid: "6560fe41305c4f72bff33aa2eeda1940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829644394b004a4289df42dafc4dcdae" is_pre_election: true
I20260808 06:57:19.862810 27889 raft_consensus.cc:2454] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6560fe41305c4f72bff33aa2eeda1940 in term 0.
I20260808 06:57:19.862885 28087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2855916982c34ef8b0e670a8704ae8ec" candidate_uuid: "6560fe41305c4f72bff33aa2eeda1940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93b0b49a6674be493397aabb106ff0d" is_pre_election: true
I20260808 06:57:19.863030 28087 raft_consensus.cc:2454] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6560fe41305c4f72bff33aa2eeda1940 in term 0.
I20260808 06:57:19.863027 27597 leader_election.cc:302] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6560fe41305c4f72bff33aa2eeda1940, 829644394b004a4289df42dafc4dcdae; no voters:
I20260808 06:57:19.863132 28181 raft_consensus.cc:2781] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:19.863209 28181 raft_consensus.cc:482] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:19.863236 28181 raft_consensus.cc:3037] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:19.863976 28181 raft_consensus.cc:504] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.864090 28181 leader_election.cc:288] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [CANDIDATE]: Term 1 election: Requested vote from peers 829644394b004a4289df42dafc4dcdae (127.21.28.66:39871), d93b0b49a6674be493397aabb106ff0d (127.21.28.67:40257)
I20260808 06:57:19.864238 28087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2855916982c34ef8b0e670a8704ae8ec" candidate_uuid: "6560fe41305c4f72bff33aa2eeda1940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93b0b49a6674be493397aabb106ff0d"
I20260808 06:57:19.864241 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2855916982c34ef8b0e670a8704ae8ec" candidate_uuid: "6560fe41305c4f72bff33aa2eeda1940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "829644394b004a4289df42dafc4dcdae"
I20260808 06:57:19.864313 27889 raft_consensus.cc:3037] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:19.864311 28087 raft_consensus.cc:3037] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:19.864989 27889 raft_consensus.cc:2454] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6560fe41305c4f72bff33aa2eeda1940 in term 1.
I20260808 06:57:19.865042 28087 raft_consensus.cc:2454] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6560fe41305c4f72bff33aa2eeda1940 in term 1.
I20260808 06:57:19.865134 27597 leader_election.cc:302] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6560fe41305c4f72bff33aa2eeda1940, 829644394b004a4289df42dafc4dcdae; no voters:
I20260808 06:57:19.865213 28181 raft_consensus.cc:2781] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:19.865360 28181 raft_consensus.cc:686] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 1 LEADER]: Becoming Leader. State: Replica: 6560fe41305c4f72bff33aa2eeda1940, State: Running, Role: LEADER
I20260808 06:57:19.865454 28181 consensus_queue.cc:237] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } }
I20260808 06:57:19.866258 27489 catalog_manager.cc:5495] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6560fe41305c4f72bff33aa2eeda1940 (127.21.28.65). New cstate: current_term: 1 leader_uuid: "6560fe41305c4f72bff33aa2eeda1940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6560fe41305c4f72bff33aa2eeda1940" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 } health_report { overall_health: UNKNOWN } } }
W20260808 06:57:19.890933 28154 tablet.cc:2369] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:19.893580 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.891498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.21.28.65:40861)
0808 06:57:19.891549 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:19.891551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:19.891559 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:19.891626 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:19.891626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:19.891641 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:19.891720 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:19.891845 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.892278 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.893037 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:19.893204 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:19.893245 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:19.893352 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:19.893448 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:19.893501 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:19.893516 (+ 15us) server_negotiation.cc:299] Negotiation successful
0808 06:57:19.893530 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:57:19.895089 28087 raft_consensus.cc:1261] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 1 FOLLOWER]: Refusing update from remote peer 6560fe41305c4f72bff33aa2eeda1940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:19.895089 27889 raft_consensus.cc:1261] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 1 FOLLOWER]: Refusing update from remote peer 6560fe41305c4f72bff33aa2eeda1940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:19.895429 28186 consensus_queue.cc:1035] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829644394b004a4289df42dafc4dcdae" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:19.895546 28181 consensus_queue.cc:1035] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93b0b49a6674be493397aabb106ff0d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 40257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:19.898689 28196 mvcc.cc:205] Tried to move back new op lower bound from 7316161494608084992 to 7316161494489485312. Current Snapshot: MvccSnapshot[applied={T|T < 7316161494608084992 or (T in {7316161494608084992})}]
I20260808 06:57:19.898980 28194 mvcc.cc:205] Tried to move back new op lower bound from 7316161494608084992 to 7316161494489485312. Current Snapshot: MvccSnapshot[applied={T|T < 7316161494608084992 or (T in {7316161494608084992})}]
I20260808 06:57:19.900039 27489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:19.900118 27489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:19.900815 27489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c36e246d4d994fab9ef20b1480c61980: Sending DeleteTablet for 3 replicas of tablet 2855916982c34ef8b0e670a8704ae8ec
I20260808 06:57:19.901163 27667 tablet_service.cc:1514] Processing DeleteTablet for tablet 2855916982c34ef8b0e670a8704ae8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:19 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:49658
I20260808 06:57:19.901142 27860 tablet_service.cc:1514] Processing DeleteTablet for tablet 2855916982c34ef8b0e670a8704ae8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:19 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:53690
I20260808 06:57:19.901274 28059 tablet_service.cc:1514] Processing DeleteTablet for tablet 2855916982c34ef8b0e670a8704ae8ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:19 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:51074
I20260808 06:57:19.901407 28207 tablet_replica.cc:330] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940: stopping tablet replica
I20260808 06:57:19.901492 28207 raft_consensus.cc:2229] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:19.901571 28207 raft_consensus.cc:2258] T 2855916982c34ef8b0e670a8704ae8ec P 6560fe41305c4f72bff33aa2eeda1940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:19.901666 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 27574
I20260808 06:57:19.901711 28208 tablet_replica.cc:330] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: stopping tablet replica
I20260808 06:57:19.901795 28208 raft_consensus.cc:2229] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:19.901798 28209 tablet_replica.cc:330] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: stopping tablet replica
I20260808 06:57:19.901906 28208 raft_consensus.cc:2258] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:19.901933 28209 raft_consensus.cc:2229] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:19.902101 28209 raft_consensus.cc:2258] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:19.902345 28208 ts_tablet_manager.cc:1905] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:19.903786 28208 ts_tablet_manager.cc:1918] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:19.903839 28208 log.cc:1196] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/2855916982c34ef8b0e670a8704ae8ec
I20260808 06:57:19.904078 28208 ts_tablet_manager.cc:1939] T 2855916982c34ef8b0e670a8704ae8ec P 829644394b004a4289df42dafc4dcdae: Deleting consensus metadata
I20260808 06:57:19.904419 27468 catalog_manager.cc:4841] TS 829644394b004a4289df42dafc4dcdae (127.21.28.66:39871): tablet 2855916982c34ef8b0e670a8704ae8ec (table test-table [id=d436d2ff9f5f4ee68345d99a1896cdec]) successfully deleted
I20260808 06:57:19.906363 28209 ts_tablet_manager.cc:1905] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:19.907560 28209 ts_tablet_manager.cc:1918] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:19.907626 28209 log.cc:1196] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/2855916982c34ef8b0e670a8704ae8ec
I20260808 06:57:19.907874 28209 ts_tablet_manager.cc:1939] T 2855916982c34ef8b0e670a8704ae8ec P d93b0b49a6674be493397aabb106ff0d: Deleting consensus metadata
I20260808 06:57:19.908218 27471 catalog_manager.cc:4841] TS d93b0b49a6674be493397aabb106ff0d (127.21.28.67:40257): tablet 2855916982c34ef8b0e670a8704ae8ec (table test-table [id=d436d2ff9f5f4ee68345d99a1896cdec]) successfully deleted
W20260808 06:57:19.924559 27469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:40861 (error 108)
I20260808 06:57:19.924737 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 27765
I20260808 06:57:19.926082 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:19.924928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40861 (local address 127.0.0.1:49688)
0808 06:57:19.926009 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:19.926053 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1058}
W20260808 06:57:19.930269 27469 catalog_manager.cc:4584] TS 6560fe41305c4f72bff33aa2eeda1940 (127.21.28.65:40861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2855916982c34ef8b0e670a8704ae8ec: Network error: Client connection negotiation failed: client connection to 127.21.28.65:40861: connect: Connection refused (error 111)
I20260808 06:57:19.936026 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 27966
I20260808 06:57:19.942358 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 27448
2026-08-08T06:57:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2139 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T06:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:19Z Disabled control of system clock
I20260808 06:57:19.970295 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:38729
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 06:57:20.048422 28216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:20.048616 28216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:20.048636 28216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:20.049718 28216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:20.049752 28216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:20.049763 28216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:20.049774 28216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:20.049791 28216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:20.051055 28216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:38729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:20.051239 28216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:20.051466 28216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:20.055967 28227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.055975 28229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:20.056038 28216 server_base.cc:1034] running on GCE node
W20260808 06:57:20.055985 28226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:20.056437 28216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:20.056684 28216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:20.057821 28216 hybrid_clock.cc:648] HybridClock initialized: now 1786172240057804 us; error 30 us; skew 500 ppm
I20260808 06:57:20.059000 28216 webserver.cc:460] Webserver started at http://127.21.28.126:34539/ using document root <none> and password file <none>
I20260808 06:57:20.059208 28216 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:20.059257 28216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:20.059379 28216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:20.060426 28216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "46fe2800a9b345d584e4197f1a671eb5"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "543c0cc55e3c775adc2ff334185e1ff2"
server_key_iv: "e90294e372c18c676dec114c016f911b"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.060745 28216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "46fe2800a9b345d584e4197f1a671eb5"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "543c0cc55e3c775adc2ff334185e1ff2"
server_key_iv: "e90294e372c18c676dec114c016f911b"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.061887 28216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:20.062510 28237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.062670 28216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:20.062763 28216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "46fe2800a9b345d584e4197f1a671eb5"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "543c0cc55e3c775adc2ff334185e1ff2"
server_key_iv: "e90294e372c18c676dec114c016f911b"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.062822 28216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:20.065958 28216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:20.066190 28216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:20.066306 28216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:20.069907 28216 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:38729
I20260808 06:57:20.069943 28307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:38729 every 8 connection(s)
I20260808 06:57:20.070225 28216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:20.070796 28308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:20.073418 28308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5: Bootstrap starting.
I20260808 06:57:20.073830 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 28216
I20260808 06:57:20.073930 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:20.073987 28308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:20.074100 21617 external_mini_cluster.cc:1428] Setting key 7e1626ef74165d70f605d91e327435d8
I20260808 06:57:20.074272 28308 log.cc:824] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:20.075126 28308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5: No bootstrap required, opened a new log
I20260808 06:57:20.076320 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.074533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.21.28.126:38729)
0808 06:57:20.074711 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.074718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.074742 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:57:20.075113 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.075119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.075151 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.075285 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.075948 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.075952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.075952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.075957 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.075994 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.076003 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.076075 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.076124 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.076176 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260808 06:57:20.076810 28308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fe2800a9b345d584e4197f1a671eb5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38729 } }
I20260808 06:57:20.076970 28308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:20.077014 28308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fe2800a9b345d584e4197f1a671eb5, State: Initialized, Role: FOLLOWER
I20260808 06:57:20.077124 28308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fe2800a9b345d584e4197f1a671eb5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38729 } }
I20260808 06:57:20.077195 28308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:20.077225 28308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:20.077270 28308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:20.078025 28308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fe2800a9b345d584e4197f1a671eb5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38729 } }
I20260808 06:57:20.078120 28308 leader_election.cc:302] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46fe2800a9b345d584e4197f1a671eb5; no voters:
I20260808 06:57:20.078265 28308 leader_election.cc:288] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:20.078299 28315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:20.078425 28315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [term 1 LEADER]: Becoming Leader. State: Replica: 46fe2800a9b345d584e4197f1a671eb5, State: Running, Role: LEADER
I20260808 06:57:20.078487 28308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:20.078522 28315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fe2800a9b345d584e4197f1a671eb5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38729 } }
I20260808 06:57:20.078817 28315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46fe2800a9b345d584e4197f1a671eb5. Latest consensus state: current_term: 1 leader_uuid: "46fe2800a9b345d584e4197f1a671eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fe2800a9b345d584e4197f1a671eb5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38729 } } }
I20260808 06:57:20.078886 28315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:20.079054 28318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46fe2800a9b345d584e4197f1a671eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fe2800a9b345d584e4197f1a671eb5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38729 } } }
I20260808 06:57:20.079114 28318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:20.079402 28326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:20.079849 28326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:20.081247 28326 catalog_manager.cc:1347] Generated new cluster ID: 3127db0ba2fa47598ff739caa2051715
I20260808 06:57:20.081291 28326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:20.089380 28326 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:20.089527 28326 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:20.093377 28326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5: Generated new TSK 0
I20260808 06:57:20.093539 28326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:57:20.101922 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38729
--builtin_ntp_servers=127.21.28.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 06:57:20.182097 28343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:20.182287 28343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:20.182307 28343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:20.183449 28343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:20.183485 28343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:20.183509 28343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:20.184810 28343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:20.185019 28343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:20.185281 28343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:20.190950 28350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.190977 28352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.190989 28356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:20.191171 28343 server_base.cc:1034] running on GCE node
I20260808 06:57:20.191411 28343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:20.191622 28343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:20.192751 28343 hybrid_clock.cc:648] HybridClock initialized: now 1786172240192737 us; error 31 us; skew 500 ppm
I20260808 06:57:20.193933 28343 webserver.cc:460] Webserver started at http://127.21.28.65:44575/ using document root <none> and password file <none>
I20260808 06:57:20.194139 28343 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:20.194186 28343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:20.194303 28343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:20.195286 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "c13ac069d7e5493a87ed717aec7bdfc3"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "9769d6c18eb70fc83c0b87fa900691fd"
server_key_iv: "bc2f8a6fe21431c7d90edd354006d0a8"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.195605 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "c13ac069d7e5493a87ed717aec7bdfc3"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "9769d6c18eb70fc83c0b87fa900691fd"
server_key_iv: "bc2f8a6fe21431c7d90edd354006d0a8"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.196898 28343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:20.197718 28365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.197878 28343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:20.197976 28343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "c13ac069d7e5493a87ed717aec7bdfc3"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "9769d6c18eb70fc83c0b87fa900691fd"
server_key_iv: "bc2f8a6fe21431c7d90edd354006d0a8"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.198057 28343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:20.201198 28343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:20.201423 28343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:20.201540 28343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:20.201726 28343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:20.202005 28343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:20.202039 28343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.202062 28343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:20.202086 28343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.207540 28343 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:38615
I20260808 06:57:20.207576 28535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:38615 every 8 connection(s)
I20260808 06:57:20.207866 28343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:20.210345 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.208505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:34167 (local address 127.21.28.126:38729)
0808 06:57:20.208557 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.208559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.208756 (+ 197us) server_negotiation.cc:407] Connection header received
0808 06:57:20.209155 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.209155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.209160 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.209211 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.210179 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.210180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.210181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.210184 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.210198 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.210202 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.210240 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.210285 (+ 45us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.210295 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 06:57:20.210538 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.208471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38729 (local address 127.21.28.65:34167)
0808 06:57:20.208643 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.208679 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.209084 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.209259 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.209262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.209270 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:20.210059 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:20.210060 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:20.210085 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:20.210111 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:20.210241 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:20.210243 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:20.210245 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:20.210290 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 06:57:20.210348 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260808 06:57:20.210965 28536 heartbeater.cc:347] Connected to a master server at 127.21.28.126:38729
I20260808 06:57:20.211042 28536 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:20.211175 28536 heartbeater.cc:511] Master 127.21.28.126:38729 requested a full tablet report, sending...
I20260808 06:57:20.211602 28262 ts_manager.cc:194] Registered new tserver with Master: c13ac069d7e5493a87ed717aec7bdfc3 (127.21.28.65:38615)
I20260808 06:57:20.216718 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 28343
I20260808 06:57:20.216794 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:20.216953 21617 external_mini_cluster.cc:1428] Setting key bd43fceba49d25e21621add0ba2cbbd7
I20260808 06:57:20.217721 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38729
--builtin_ntp_servers=127.21.28.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 06:57:20.296698 28541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:20.296880 28541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:20.296901 28541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:20.298065 28541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:20.298105 28541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:20.298131 28541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:20.299425 28541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:20.299621 28541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:20.299889 28541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:20.304610 28549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.304621 28554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.304647 28550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:20.304915 28541 server_base.cc:1034] running on GCE node
I20260808 06:57:20.305121 28541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:20.305310 28541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:20.306439 28541 hybrid_clock.cc:648] HybridClock initialized: now 1786172240306431 us; error 26 us; skew 500 ppm
I20260808 06:57:20.307545 28541 webserver.cc:460] Webserver started at http://127.21.28.66:40607/ using document root <none> and password file <none>
I20260808 06:57:20.307767 28541 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:20.307814 28541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:20.307936 28541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:20.308992 28541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "817e901ffa3659f72c25f3be56241a9c"
server_key_iv: "5234be9fdc867affdf2a159944fab745"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.309307 28541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "817e901ffa3659f72c25f3be56241a9c"
server_key_iv: "5234be9fdc867affdf2a159944fab745"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.310482 28541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:20.311077 28560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.311205 28541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:20.311287 28541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "817e901ffa3659f72c25f3be56241a9c"
server_key_iv: "5234be9fdc867affdf2a159944fab745"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.311352 28541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:20.314584 28541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:20.314805 28541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:20.314916 28541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:20.315088 28541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:20.315372 28541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:20.315405 28541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.315431 28541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:20.315456 28541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.321040 28541 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:42561
I20260808 06:57:20.321074 28723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:42561 every 8 connection(s)
I20260808 06:57:20.321349 28541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:20.322187 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 28541
I20260808 06:57:20.322273 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:20.322436 21617 external_mini_cluster.cc:1428] Setting key ab54ba35d01c73dd060fd9947c0e30b6
I20260808 06:57:20.323277 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38729
--builtin_ntp_servers=127.21.28.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 06:57:20.324083 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.321923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:39053 (local address 127.21.28.126:38729)
0808 06:57:20.321969 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.321971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.322184 (+ 213us) server_negotiation.cc:407] Connection header received
0808 06:57:20.322676 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.322678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.322684 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.322730 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.323856 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.323858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.323858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.323863 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.323878 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.323882 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.323930 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.324001 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.324014 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:57:20.324256 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.321904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38729 (local address 127.21.28.66:39053)
0808 06:57:20.322069 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.322105 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.322580 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.322803 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.322807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.322816 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:20.323720 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:20.323720 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:20.323753 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:20.323782 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:20.323937 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:20.323942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:20.323945 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:20.324011 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 06:57:20.324067 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260808 06:57:20.324668 28726 heartbeater.cc:347] Connected to a master server at 127.21.28.126:38729
I20260808 06:57:20.324775 28726 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:20.324918 28726 heartbeater.cc:511] Master 127.21.28.126:38729 requested a full tablet report, sending...
I20260808 06:57:20.325259 28262 ts_manager.cc:194] Registered new tserver with Master: 3b2288cc181f4ca0a2a7e768f40f7ea9 (127.21.28.66:42561)
W20260808 06:57:20.402613 28729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:20.402794 28729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:20.402814 28729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:20.403918 28729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:20.403951 28729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:20.403975 28729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:20.405225 28729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:20.405391 28729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:20.405609 28729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:20.410264 28738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.410274 28743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:20.410454 28729 server_base.cc:1034] running on GCE node
W20260808 06:57:20.410311 28739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:20.410668 28729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:20.410892 28729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:20.412026 28729 hybrid_clock.cc:648] HybridClock initialized: now 1786172240412014 us; error 28 us; skew 500 ppm
I20260808 06:57:20.413249 28729 webserver.cc:460] Webserver started at http://127.21.28.67:45221/ using document root <none> and password file <none>
I20260808 06:57:20.413450 28729 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:20.413497 28729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:20.413664 28729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:20.414647 28729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "e03c9b19d35649cda36d1e3d9ae02feb"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "a0358057352cc76e268e05b3af41a88b"
server_key_iv: "3996ab220e82881eec292a4cf7d8d1db"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.414990 28729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "e03c9b19d35649cda36d1e3d9ae02feb"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "a0358057352cc76e268e05b3af41a88b"
server_key_iv: "3996ab220e82881eec292a4cf7d8d1db"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.416230 28729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:20.416848 28754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.416991 28729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:20.417073 28729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "e03c9b19d35649cda36d1e3d9ae02feb"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "a0358057352cc76e268e05b3af41a88b"
server_key_iv: "3996ab220e82881eec292a4cf7d8d1db"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.417135 28729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:20.420383 28729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:20.420596 28729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:20.420709 28729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:20.420893 28729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:20.421150 28729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:20.421183 28729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.421207 28729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:20.421239 28729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.426697 28729 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:35279
I20260808 06:57:20.426733 28929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:35279 every 8 connection(s)
I20260808 06:57:20.426993 28729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:20.427947 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 28729
I20260808 06:57:20.428056 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:20.428233 21617 external_mini_cluster.cc:1428] Setting key 8a1faa7d1f06ed440ca42f99856b82a1
I20260808 06:57:20.429804 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.427594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:42337 (local address 127.21.28.126:38729)
0808 06:57:20.427698 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.427700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.427881 (+ 181us) server_negotiation.cc:407] Connection header received
0808 06:57:20.428433 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.428434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.428441 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.428502 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.429602 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.429603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.429603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.429607 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.429622 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.429626 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.429667 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.429742 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.429753 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 06:57:20.429939 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.427566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38729 (local address 127.21.28.67:42337)
0808 06:57:20.427761 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.427799 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.428329 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.428548 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.428551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.428559 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:20.429475 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:20.429475 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:20.429499 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:20.429525 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:20.429674 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:20.429676 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:20.429679 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:20.429725 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 06:57:20.429768 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260808 06:57:20.430344 28931 heartbeater.cc:347] Connected to a master server at 127.21.28.126:38729
I20260808 06:57:20.430425 28931 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:20.430531 28931 heartbeater.cc:511] Master 127.21.28.126:38729 requested a full tablet report, sending...
I20260808 06:57:20.430820 28262 ts_manager.cc:194] Registered new tserver with Master: e03c9b19d35649cda36d1e3d9ae02feb (127.21.28.67:35279)
I20260808 06:57:20.439639 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:57:20.441736 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.440855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.21.28.126:38729)
0808 06:57:20.440921 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.440923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.441000 (+ 77us) server_negotiation.cc:407] Connection header received
0808 06:57:20.441059 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.441060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.441065 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.441116 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.441557 (+ 441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.441558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.441558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.441562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.441575 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.441579 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.441622 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.441661 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.441674 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:57:20.443025 28262 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:20.444572 28262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:20.447922 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.446354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35279 (local address 127.0.0.1:39664)
0808 06:57:20.446509 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.446548 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.446599 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.447015 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.447021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.447027 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:20.447509 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:20.447509 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:20.447525 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:20.447547 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:20.447781 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:20.447787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:20.447789 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:20.447844 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 06:57:20.447858 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":50,"threads_started":1}
I20260808 06:57:20.448083 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.446757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.21.28.65:38615)
0808 06:57:20.446932 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.446936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.446944 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:20.446967 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.446971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.446980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.447048 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.447764 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.447771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.447772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.447776 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.447808 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.447834 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.447882 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.447978 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.447997 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260808 06:57:20.448107 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.446488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.21.28.67:35279)
0808 06:57:20.446654 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.446658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.446666 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:20.446685 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.446687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.446694 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.446759 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.447617 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.447625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.447625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.447629 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.447662 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.447704 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.447756 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.448022 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.448039 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260808 06:57:20.448115 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.446535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:38615 (local address 127.0.0.1:53202)
0808 06:57:20.446749 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.446757 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.446788 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.447315 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.447316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.447318 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:20.447638 (+ 320us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:20.447638 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:20.447645 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:20.447656 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:20.447904 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:20.447905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:20.447906 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:20.447960 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 06:57:20.447971 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":72,"threads_started":1}
I20260808 06:57:20.448781 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.446356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:42561 (local address 127.0.0.1:60152)
0808 06:57:20.446638 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.446648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.446690 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.447035 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.447036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.447038 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:20.447809 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:20.447810 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:20.447816 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:20.447826 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:20.448656 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:20.448657 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:20.448658 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:20.448712 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 06:57:20.448725 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":88,"threads_started":1}
I20260808 06:57:20.448719 28824 tablet_service.cc:1467] Processing CreateTablet for tablet 8d68b4773f7449d98262fb94d65c6dee (DEFAULT_TABLE table=test-table [id=187f3784ff134e8391e8f582bb604ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:20.448951 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.446465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.21.28.66:42561)
0808 06:57:20.446886 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.446890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.446900 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:20.446935 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.446937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.446947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.447030 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.447915 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.447922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.447923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.447927 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.448103 (+ 176us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.448582 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.448652 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.448721 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.448738 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":78,"threads_started":1}
I20260808 06:57:20.448942 28433 tablet_service.cc:1467] Processing CreateTablet for tablet 8d68b4773f7449d98262fb94d65c6dee (DEFAULT_TABLE table=test-table [id=187f3784ff134e8391e8f582bb604ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:20.449064 28824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d68b4773f7449d98262fb94d65c6dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:20.449254 28433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d68b4773f7449d98262fb94d65c6dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:20.449704 28627 tablet_service.cc:1467] Processing CreateTablet for tablet 8d68b4773f7449d98262fb94d65c6dee (DEFAULT_TABLE table=test-table [id=187f3784ff134e8391e8f582bb604ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:20.450024 28627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d68b4773f7449d98262fb94d65c6dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:20.451514 28947 tablet_bootstrap.cc:492] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: Bootstrap starting.
I20260808 06:57:20.452279 28947 tablet_bootstrap.cc:654] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:20.452409 28948 tablet_bootstrap.cc:492] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3: Bootstrap starting.
I20260808 06:57:20.452593 28947 log.cc:824] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:20.452991 28948 tablet_bootstrap.cc:654] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:20.453073 28951 tablet_bootstrap.cc:492] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: Bootstrap starting.
I20260808 06:57:20.453258 28948 log.cc:824] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:20.453626 28951 tablet_bootstrap.cc:654] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:20.453735 28947 tablet_bootstrap.cc:492] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: No bootstrap required, opened a new log
I20260808 06:57:20.453797 28947 ts_tablet_manager.cc:1397] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:20.453886 28951 log.cc:824] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:20.454481 28951 tablet_bootstrap.cc:492] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: No bootstrap required, opened a new log
I20260808 06:57:20.454530 28951 ts_tablet_manager.cc:1397] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:20.455642 28947 raft_consensus.cc:348] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.455804 28947 raft_consensus.cc:374] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:20.455849 28947 raft_consensus.cc:729] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e03c9b19d35649cda36d1e3d9ae02feb, State: Initialized, Role: FOLLOWER
I20260808 06:57:20.455965 28947 consensus_queue.cc:260] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.456209 28947 ts_tablet_manager.cc:1428] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 06:57:20.456235 28931 heartbeater.cc:503] Master 127.21.28.126:38729 was elected leader, sending a full tablet report...
I20260808 06:57:20.456491 28951 raft_consensus.cc:348] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.456642 28951 raft_consensus.cc:374] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:20.456684 28951 raft_consensus.cc:729] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2288cc181f4ca0a2a7e768f40f7ea9, State: Initialized, Role: FOLLOWER
I20260808 06:57:20.456785 28948 tablet_bootstrap.cc:492] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3: No bootstrap required, opened a new log
I20260808 06:57:20.456789 28951 consensus_queue.cc:260] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.456847 28948 ts_tablet_manager.cc:1397] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 06:57:20.457044 28951 ts_tablet_manager.cc:1428] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 06:57:20.457082 28726 heartbeater.cc:503] Master 127.21.28.126:38729 was elected leader, sending a full tablet report...
I20260808 06:57:20.459082 28948 raft_consensus.cc:348] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.459197 28948 raft_consensus.cc:374] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:20.459216 28948 raft_consensus.cc:729] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13ac069d7e5493a87ed717aec7bdfc3, State: Initialized, Role: FOLLOWER
I20260808 06:57:20.459296 28948 consensus_queue.cc:260] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.459520 28948 ts_tablet_manager.cc:1428] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 06:57:20.459539 28536 heartbeater.cc:503] Master 127.21.28.126:38729 was elected leader, sending a full tablet report...
I20260808 06:57:20.534819 28959 raft_consensus.cc:482] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:20.534976 28959 raft_consensus.cc:504] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.535328 28959 leader_election.cc:288] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b2288cc181f4ca0a2a7e768f40f7ea9 (127.21.28.66:42561), e03c9b19d35649cda36d1e3d9ae02feb (127.21.28.67:35279)
I20260808 06:57:20.536535 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.535497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:42561 (local address 127.21.28.65:37845)
0808 06:57:20.535574 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.535586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.535624 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.535790 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.535791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.535793 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:20.536231 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:20.536232 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:20.536239 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:20.536250 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:20.536403 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:20.536404 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:20.536405 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:20.536450 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 06:57:20.536464 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 06:57:20.536535 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.535566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:37845 (local address 127.21.28.66:42561)
0808 06:57:20.535612 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.535614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.535625 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:57:20.535704 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.535705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.535712 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.535773 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.536327 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.536328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.536329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.536334 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.536352 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.536356 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.536402 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.536452 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.536464 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:20.536705 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.535604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:36893 (local address 127.21.28.67:35279)
0808 06:57:20.535646 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.535648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.535768 (+ 120us) server_negotiation.cc:407] Connection header received
0808 06:57:20.535816 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.535817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.535823 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.535880 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.536455 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.536457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.536457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.536462 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.536477 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.536482 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.536531 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.536623 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.536636 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:20.536921 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.535505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35279 (local address 127.21.28.65:36893)
0808 06:57:20.535699 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.535709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.535762 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.535888 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.535888 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.535890 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:20.536347 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:20.536348 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:20.536354 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:20.536365 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:20.536558 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:20.536559 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:20.536560 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:20.536786 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 06:57:20.536801 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260808 06:57:20.536926 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d68b4773f7449d98262fb94d65c6dee" candidate_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" is_pre_election: true
I20260808 06:57:20.537081 28654 raft_consensus.cc:2454] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c13ac069d7e5493a87ed717aec7bdfc3 in term 0.
I20260808 06:57:20.537281 28850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d68b4773f7449d98262fb94d65c6dee" candidate_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" is_pre_election: true
I20260808 06:57:20.537357 28366 leader_election.cc:302] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b2288cc181f4ca0a2a7e768f40f7ea9, c13ac069d7e5493a87ed717aec7bdfc3; no voters:
I20260808 06:57:20.537451 28850 raft_consensus.cc:2454] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c13ac069d7e5493a87ed717aec7bdfc3 in term 0.
I20260808 06:57:20.537482 28959 raft_consensus.cc:2781] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:20.537556 28959 raft_consensus.cc:482] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:20.537578 28959 raft_consensus.cc:3037] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:20.538292 28959 raft_consensus.cc:504] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.538414 28959 leader_election.cc:288] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [CANDIDATE]: Term 1 election: Requested vote from peers 3b2288cc181f4ca0a2a7e768f40f7ea9 (127.21.28.66:42561), e03c9b19d35649cda36d1e3d9ae02feb (127.21.28.67:35279)
I20260808 06:57:20.538558 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d68b4773f7449d98262fb94d65c6dee" candidate_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9"
I20260808 06:57:20.538563 28850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d68b4773f7449d98262fb94d65c6dee" candidate_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e03c9b19d35649cda36d1e3d9ae02feb"
I20260808 06:57:20.538630 28850 raft_consensus.cc:3037] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:20.538633 28654 raft_consensus.cc:3037] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:20.539351 28850 raft_consensus.cc:2454] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c13ac069d7e5493a87ed717aec7bdfc3 in term 1.
I20260808 06:57:20.539361 28654 raft_consensus.cc:2454] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c13ac069d7e5493a87ed717aec7bdfc3 in term 1.
I20260808 06:57:20.539511 28366 leader_election.cc:302] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b2288cc181f4ca0a2a7e768f40f7ea9, c13ac069d7e5493a87ed717aec7bdfc3; no voters:
I20260808 06:57:20.539605 28959 raft_consensus.cc:2781] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:20.539793 28959 raft_consensus.cc:686] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [term 1 LEADER]: Becoming Leader. State: Replica: c13ac069d7e5493a87ed717aec7bdfc3, State: Running, Role: LEADER
I20260808 06:57:20.539880 28959 consensus_queue.cc:237] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } }
I20260808 06:57:20.540668 28261 catalog_manager.cc:5495] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c13ac069d7e5493a87ed717aec7bdfc3 (127.21.28.65). New cstate: current_term: 1 leader_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c13ac069d7e5493a87ed717aec7bdfc3" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38615 } health_report { overall_health: HEALTHY } } }
I20260808 06:57:20.562572 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.561702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.21.28.65:38615)
0808 06:57:20.561757 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.561759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.561768 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:20.561813 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.561814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.561826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.561868 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.562368 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.562369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.562369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.562373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.562387 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.562390 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.562444 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.562496 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.562507 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:57:20.564109 28850 raft_consensus.cc:1261] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 1 FOLLOWER]: Refusing update from remote peer c13ac069d7e5493a87ed717aec7bdfc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:20.564110 28654 raft_consensus.cc:1261] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 1 FOLLOWER]: Refusing update from remote peer c13ac069d7e5493a87ed717aec7bdfc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:20.564380 28959 consensus_queue.cc:1035] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b2288cc181f4ca0a2a7e768f40f7ea9" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:20.564510 28964 consensus_queue.cc:1035] T 8d68b4773f7449d98262fb94d65c6dee P c13ac069d7e5493a87ed717aec7bdfc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03c9b19d35649cda36d1e3d9ae02feb" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:20.567389 28970 mvcc.cc:205] Tried to move back new op lower bound from 7316161497348296704 to 7316161497251893248. Current Snapshot: MvccSnapshot[applied={T|T < 7316161497348296704 or (T in {7316161497348296704})}]
I20260808 06:57:20.568974 28261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:20.569052 28261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:20.569736 28261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46fe2800a9b345d584e4197f1a671eb5: Sending DeleteTablet for 3 replicas of tablet 8d68b4773f7449d98262fb94d65c6dee
I20260808 06:57:20.570056 28433 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d68b4773f7449d98262fb94d65c6dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:20 UTC) from {username='slave'} at 127.0.0.1:53202
I20260808 06:57:20.570075 28627 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d68b4773f7449d98262fb94d65c6dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:20 UTC) from {username='slave'} at 127.0.0.1:60152
I20260808 06:57:20.570262 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 28343
I20260808 06:57:20.570250 28824 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d68b4773f7449d98262fb94d65c6dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:20 UTC) from {username='slave'} at 127.0.0.1:39664
I20260808 06:57:20.576411 28985 tablet_replica.cc:330] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: stopping tablet replica
I20260808 06:57:20.576519 28985 raft_consensus.cc:2229] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:20.576613 28985 raft_consensus.cc:2258] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:20.576767 28984 tablet_replica.cc:330] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: stopping tablet replica
I20260808 06:57:20.576858 28984 raft_consensus.cc:2229] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:20.576960 28984 raft_consensus.cc:2258] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:20.577140 28985 ts_tablet_manager.cc:1905] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:20.577363 28984 ts_tablet_manager.cc:1905] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:20.578624 28985 ts_tablet_manager.cc:1918] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:20.578692 28985 log.cc:1196] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/8d68b4773f7449d98262fb94d65c6dee
I20260808 06:57:20.578905 28985 ts_tablet_manager.cc:1939] T 8d68b4773f7449d98262fb94d65c6dee P e03c9b19d35649cda36d1e3d9ae02feb: Deleting consensus metadata
I20260808 06:57:20.578960 28984 ts_tablet_manager.cc:1918] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:20.579026 28984 log.cc:1196] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/8d68b4773f7449d98262fb94d65c6dee
I20260808 06:57:20.579241 28984 ts_tablet_manager.cc:1939] T 8d68b4773f7449d98262fb94d65c6dee P 3b2288cc181f4ca0a2a7e768f40f7ea9: Deleting consensus metadata
I20260808 06:57:20.579272 28243 catalog_manager.cc:4841] TS e03c9b19d35649cda36d1e3d9ae02feb (127.21.28.67:35279): tablet 8d68b4773f7449d98262fb94d65c6dee (table test-table [id=187f3784ff134e8391e8f582bb604ad5]) successfully deleted
I20260808 06:57:20.579579 28238 catalog_manager.cc:4841] TS 3b2288cc181f4ca0a2a7e768f40f7ea9 (127.21.28.66:42561): tablet 8d68b4773f7449d98262fb94d65c6dee (table test-table [id=187f3784ff134e8391e8f582bb604ad5]) successfully deleted
W20260808 06:57:20.580729 28243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:38615 (error 108)
W20260808 06:57:20.581211 28940 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:20.581100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:38615 (local address 127.0.0.1:53220)
0808 06:57:20.581146 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.581181 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:38615: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 06:57:20.581334 28243 catalog_manager.cc:4584] TS c13ac069d7e5493a87ed717aec7bdfc3 (127.21.28.65:38615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d68b4773f7449d98262fb94d65c6dee: Network error: Client connection negotiation failed: client connection to 127.21.28.65:38615: connect: Connection reset by peer (error 104)
I20260808 06:57:20.581652 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 28541
I20260808 06:57:20.590973 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 28729
I20260808 06:57:20.601864 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 28216
2026-08-08T06:57:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (661 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 06:57:20.625181 21617 mini_oidc.cc:152] Starting JWKS server
I20260808 06:57:20.626338 21617 webserver.cc:460] Webserver started at https://127.0.0.1:43937/ using document root <none> and password file <none>
I20260808 06:57:20.626417 21617 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 06:57:20.626573 21617 webserver.cc:460] Webserver started at http://127.0.0.1:46639/ using document root <none> and password file <none>
2026-08-08T06:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:20Z Disabled control of system clock
I20260808 06:57:20.633816 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46843
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:42203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46843
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43937/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/testchainca.cert with env {}
W20260808 06:57:20.710815 28992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:20.711014 28992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:20.711045 28992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:20.712415 28992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:20.712466 28992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:20.712483 28992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:20.712498 28992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:20.712513 28992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 06:57:20.712531 28992 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 06:57:20.712554 28992 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43937/jwks.json
I20260808 06:57:20.714177 28992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43937/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:20.714406 28992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:20.714597 28992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:20.718978 29001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.718993 29006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.718993 29002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:20.719211 28992 server_base.cc:1034] running on GCE node
I20260808 06:57:20.719475 28992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:20.719722 28992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:20.720856 28992 hybrid_clock.cc:648] HybridClock initialized: now 1786172240720847 us; error 27 us; skew 500 ppm
I20260808 06:57:20.721992 28992 webserver.cc:460] Webserver started at http://127.21.28.126:34827/ using document root <none> and password file <none>
I20260808 06:57:20.722198 28992 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:20.722241 28992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:20.722357 28992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:20.723308 28992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "708fd8337d3045dd918c5cabe408fccd"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "8dbfaa35bcf04358c2434d7791748d1e"
server_key_iv: "def4926c95e97a1e0968fd1d9b624aef"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.723630 28992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "708fd8337d3045dd918c5cabe408fccd"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "8dbfaa35bcf04358c2434d7791748d1e"
server_key_iv: "def4926c95e97a1e0968fd1d9b624aef"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.724789 28992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:20.725418 29012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.725569 28992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:20.725649 28992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "708fd8337d3045dd918c5cabe408fccd"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "8dbfaa35bcf04358c2434d7791748d1e"
server_key_iv: "def4926c95e97a1e0968fd1d9b624aef"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.725708 28992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:20.738057 28992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:20.738307 28992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:20.738425 28992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:20.742089 28992 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46843
I20260808 06:57:20.742118 29082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46843 every 8 connection(s)
I20260808 06:57:20.742408 28992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:20.743356 29084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:20.745584 29084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd: Bootstrap starting.
I20260808 06:57:20.746143 29084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:20.746417 29084 log.cc:824] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:20.747109 29084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd: No bootstrap required, opened a new log
I20260808 06:57:20.747372 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 28992
I20260808 06:57:20.747470 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:20.747659 21617 external_mini_cluster.cc:1428] Setting key a795801f96da6972e869675dbb5ea734
I20260808 06:57:20.749220 29084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708fd8337d3045dd918c5cabe408fccd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46843 } }
I20260808 06:57:20.749358 29084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:20.749384 29084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708fd8337d3045dd918c5cabe408fccd, State: Initialized, Role: FOLLOWER
I20260808 06:57:20.749547 29084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708fd8337d3045dd918c5cabe408fccd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46843 } }
I20260808 06:57:20.749631 29084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:20.749672 29084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:20.749714 29084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:20.750243 29084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708fd8337d3045dd918c5cabe408fccd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46843 } }
I20260808 06:57:20.750339 29084 leader_election.cc:302] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 708fd8337d3045dd918c5cabe408fccd; no voters:
I20260808 06:57:20.750481 29084 leader_election.cc:288] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:20.750530 29095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:20.750663 29095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [term 1 LEADER]: Becoming Leader. State: Replica: 708fd8337d3045dd918c5cabe408fccd, State: Running, Role: LEADER
I20260808 06:57:20.750721 29084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:20.750748 29095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708fd8337d3045dd918c5cabe408fccd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46843 } }
I20260808 06:57:20.751103 29095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 708fd8337d3045dd918c5cabe408fccd. Latest consensus state: current_term: 1 leader_uuid: "708fd8337d3045dd918c5cabe408fccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708fd8337d3045dd918c5cabe408fccd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46843 } } }
I20260808 06:57:20.751176 29095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:20.751330 29096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "708fd8337d3045dd918c5cabe408fccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708fd8337d3045dd918c5cabe408fccd" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46843 } } }
I20260808 06:57:20.751389 29096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:20.751503 29104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:20.752224 29104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:20.753748 29104 catalog_manager.cc:1347] Generated new cluster ID: 401099c160d54523ba444c8359a79035
I20260808 06:57:20.753801 29104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:20.754033 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.748197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.21.28.126:46843)
0808 06:57:20.748420 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.748427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.748453 (+ 26us) server_negotiation.cc:407] Connection header received
0808 06:57:20.749215 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.749223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.749284 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.749438 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.749666 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:20.751094 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:20.752215 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:20.752722 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:20.752844 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:20.753448 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.753451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.753451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.753601 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.753633 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.753643 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.753699 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.753802 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.753875 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260808 06:57:20.754856 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:57:20.769639 29104 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:20.770465 29104 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:20.775802 29104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 708fd8337d3045dd918c5cabe408fccd: Generated new TSK 0
I20260808 06:57:20.776093 29104 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 06:57:20.810145 29133 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 06:57:20.810274 29133 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 06:57:20.810370 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.787578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.21.28.126:46843)
0808 06:57:20.787650 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.787652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.787820 (+ 168us) server_negotiation.cc:407] Connection header received
0808 06:57:20.787986 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.787987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.788006 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.788090 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0808 06:57:20.788331 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:20.788798 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:20.789882 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:20.790059 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:20.790098 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:20.790290 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 06:57:20.810162 (+ 19872us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 06:57:20.810306 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 06:57:20.810427 29094 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 06:57:20.810822 29132 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:20.787406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46843 (local address 127.0.0.1:40096)
0808 06:57:20.787729 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:20.787740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:20.787845 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:20.788124 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:20.788125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:20.788128 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0808 06:57:20.788253 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:20.788257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:20.788864 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:20.788865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:20.789812 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:20.789816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:20.790115 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:20.790116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:20.790182 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:20.790203 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 06:57:20.810295 (+ 20092us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 06:57:20.810487 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.28.126:46843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":7,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.28.126:46843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 06:57:20.811196 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 28992
2026-08-08T06:57:20Z chronyd exiting
I20260808 06:57:20.859982 21617 test_util.cc:175] -----------------------------------------------
I20260808 06:57:20.860054 21617 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786172218768454-21617-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 06:57:20.860822 21617 mini_oidc.cc:152] Starting JWKS server
I20260808 06:57:20.861472 21617 webserver.cc:460] Webserver started at https://127.0.0.1:33235/ using document root <none> and password file <none>
I20260808 06:57:20.861514 21617 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 06:57:20.861649 21617 webserver.cc:460] Webserver started at http://127.0.0.1:34437/ using document root <none> and password file <none>
2026-08-08T06:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:20Z Disabled control of system clock
I20260808 06:57:20.868997 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:44345
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:44345
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33235/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/testchainca.cert with env {}
W20260808 06:57:20.948501 29140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:20.948694 29140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:20.948732 29140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:20.949855 29140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:20.949893 29140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:20.949910 29140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:20.949926 29140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:20.949942 29140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 06:57:20.949962 29140 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 06:57:20.949978 29140 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33235/jwks.json
I20260808 06:57:20.951449 29140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:44345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33235/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:20.951714 29140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:20.951911 29140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:20.956357 29151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.956363 29147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:20.956400 29146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:20.956642 29140 server_base.cc:1034] running on GCE node
I20260808 06:57:20.956874 29140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:20.957137 29140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:20.958297 29140 hybrid_clock.cc:648] HybridClock initialized: now 1786172240958277 us; error 35 us; skew 500 ppm
I20260808 06:57:20.962461 29140 webserver.cc:460] Webserver started at http://127.21.28.126:33563/ using document root <none> and password file <none>
I20260808 06:57:20.962687 29140 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:20.962761 29140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:20.962891 29140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:20.963919 29140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "799e4dee7345484e97f8016ab7dda80c"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "d5db2bb15b0ad9289e0ed52a34c58b29"
server_key_iv: "96e68caea9e626eb3a3daa759767c9a1"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.964241 29140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "799e4dee7345484e97f8016ab7dda80c"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "d5db2bb15b0ad9289e0ed52a34c58b29"
server_key_iv: "96e68caea9e626eb3a3daa759767c9a1"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.965442 29140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:20.966090 29164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:20.966234 29140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:20.966317 29140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "799e4dee7345484e97f8016ab7dda80c"
format_stamp: "Formatted at 2026-08-08 06:57:20 on dist-test-slave-nj21"
server_key: "d5db2bb15b0ad9289e0ed52a34c58b29"
server_key_iv: "96e68caea9e626eb3a3daa759767c9a1"
server_key_version: "encryptionkey@0"
I20260808 06:57:20.966380 29140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:20.979728 29140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:20.979992 29140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:20.980113 29140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:20.983731 29140 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:44345
I20260808 06:57:20.983758 29233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:44345 every 8 connection(s)
I20260808 06:57:20.984058 29140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:20.984653 29235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:20.987298 29235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c: Bootstrap starting.
I20260808 06:57:20.987855 29235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:20.988107 29235 log.cc:824] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:20.988688 29235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c: No bootstrap required, opened a new log
I20260808 06:57:20.990203 29235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e4dee7345484e97f8016ab7dda80c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44345 } }
I20260808 06:57:20.990329 29235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:20.990360 29235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 799e4dee7345484e97f8016ab7dda80c, State: Initialized, Role: FOLLOWER
I20260808 06:57:20.990497 29235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e4dee7345484e97f8016ab7dda80c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44345 } }
I20260808 06:57:20.990561 29235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:20.990583 29235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:20.990628 29235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:20.991111 29235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e4dee7345484e97f8016ab7dda80c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44345 } }
I20260808 06:57:20.991199 29235 leader_election.cc:302] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 799e4dee7345484e97f8016ab7dda80c; no voters:
I20260808 06:57:20.991346 29235 leader_election.cc:288] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:20.991387 29238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:20.991528 29238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [term 1 LEADER]: Becoming Leader. State: Replica: 799e4dee7345484e97f8016ab7dda80c, State: Running, Role: LEADER
I20260808 06:57:20.991568 29235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:20.991624 29238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e4dee7345484e97f8016ab7dda80c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44345 } }
I20260808 06:57:20.991990 29240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "799e4dee7345484e97f8016ab7dda80c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e4dee7345484e97f8016ab7dda80c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44345 } } }
I20260808 06:57:20.992077 29240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:20.992158 29242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 799e4dee7345484e97f8016ab7dda80c. Latest consensus state: current_term: 1 leader_uuid: "799e4dee7345484e97f8016ab7dda80c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e4dee7345484e97f8016ab7dda80c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44345 } } }
I20260808 06:57:20.992237 29242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:20.992614 29248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:20.992761 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 29140
I20260808 06:57:20.992856 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:20.993053 21617 external_mini_cluster.cc:1428] Setting key fff1019b7120f302b424ff001eefa103
I20260808 06:57:20.993078 29248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:20.994617 29248 catalog_manager.cc:1347] Generated new cluster ID: 2aeb1fe0790d451d9ca5e5a07cfd4de7
I20260808 06:57:20.994669 29248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:20.998878 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:20.993574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.21.28.126:44345)
0808 06:57:20.993747 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:20.993755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:20.993786 (+ 31us) server_negotiation.cc:407] Connection header received
0808 06:57:20.993899 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:20.993906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:20.993970 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:20.994102 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:20.994345 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:20.995664 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:20.996763 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:20.997342 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:20.997455 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:20.998095 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:20.998098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:20.998099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:20.998289 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:20.998329 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:20.998343 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:20.998405 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:20.998618 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 06:57:20.998688 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260808 06:57:20.999570 29248 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:20.999783 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:57:21.000851 29248 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:21.005028 29248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 799e4dee7345484e97f8016ab7dda80c: Generated new TSK 0
I20260808 06:57:21.005182 29248 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 06:57:21.029712 29280 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 06:57:21.029770 29280 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260808 06:57:21.030133 29277 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:21.016044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:44345 (local address 127.0.0.1:35256)
0808 06:57:21.016205 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:21.016213 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:21.016261 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:21.016382 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:21.016382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:21.016384 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 06:57:21.016432 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:21.016433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.017331 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.017331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:21.017894 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:21.017896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.018346 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.018346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:21.018381 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:21.018386 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 06:57:21.029978 (+ 11592us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 06:57:21.030064 (+ 86us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.28.126:44345: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260808 06:57:21.030200 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:21.016122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.21.28.126:44345)
0808 06:57:21.016166 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:21.016169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:21.016266 (+ 97us) server_negotiation.cc:407] Connection header received
0808 06:57:21.016301 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:21.016302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:21.016313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:21.016384 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0808 06:57:21.016482 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.017250 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.017961 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.018282 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.018345 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:21.018469 (+ 124us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 06:57:21.029893 (+ 11424us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 06:57:21.030134 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35256: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 06:57:21.030243 29266 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35256: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33235/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260808 06:57:21.030584 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 29140
2026-08-08T06:57:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 06:57:21.074080 21617 mini_oidc.cc:152] Starting JWKS server
I20260808 06:57:21.074710 21617 webserver.cc:460] Webserver started at https://127.0.0.1:38339/ using document root <none> and password file <none>
I20260808 06:57:21.074750 21617 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 06:57:21.074857 21617 webserver.cc:460] Webserver started at http://127.0.0.1:37225/ using document root <none> and password file <none>
2026-08-08T06:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:21Z Disabled control of system clock
I20260808 06:57:21.082413 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:40811
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:40811
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38339/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 06:57:21.162107 29289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:21.162304 29289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:21.162334 29289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:21.163445 29289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:21.163484 29289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:21.163501 29289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:21.163517 29289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:21.163532 29289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 06:57:21.163546 29289 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 06:57:21.163565 29289 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38339/jwks.json
I20260808 06:57:21.164921 29289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:40811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:40811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38339/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:21.165158 29289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:21.165407 29289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:21.169925 29298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:21.169937 29300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:21.169937 29297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:21.170149 29289 server_base.cc:1034] running on GCE node
I20260808 06:57:21.170362 29289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:21.170614 29289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:21.171717 29289 hybrid_clock.cc:648] HybridClock initialized: now 1786172241171704 us; error 27 us; skew 500 ppm
I20260808 06:57:21.172868 29289 webserver.cc:460] Webserver started at http://127.21.28.126:35471/ using document root <none> and password file <none>
I20260808 06:57:21.173075 29289 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:21.173121 29289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:21.173239 29289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:21.174202 29289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "6ffac90d0dfe424e974b3ee0cd94a35a"
format_stamp: "Formatted at 2026-08-08 06:57:21 on dist-test-slave-nj21"
server_key: "6388f888033d4f7fc8beb840b1b1b5e9"
server_key_iv: "3f55eebde6d48b0df193e7bb9b45b072"
server_key_version: "encryptionkey@0"
I20260808 06:57:21.174546 29289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "6ffac90d0dfe424e974b3ee0cd94a35a"
format_stamp: "Formatted at 2026-08-08 06:57:21 on dist-test-slave-nj21"
server_key: "6388f888033d4f7fc8beb840b1b1b5e9"
server_key_iv: "3f55eebde6d48b0df193e7bb9b45b072"
server_key_version: "encryptionkey@0"
I20260808 06:57:21.175766 29289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 06:57:21.176432 29315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:21.176555 29289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:21.176632 29289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "6ffac90d0dfe424e974b3ee0cd94a35a"
format_stamp: "Formatted at 2026-08-08 06:57:21 on dist-test-slave-nj21"
server_key: "6388f888033d4f7fc8beb840b1b1b5e9"
server_key_iv: "3f55eebde6d48b0df193e7bb9b45b072"
server_key_version: "encryptionkey@0"
I20260808 06:57:21.176698 29289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:21.191565 29289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:21.191835 29289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:21.191964 29289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:21.195519 29289 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:40811
I20260808 06:57:21.195549 29389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:40811 every 8 connection(s)
I20260808 06:57:21.195858 29289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:21.196593 29390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:21.199312 29390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a: Bootstrap starting.
I20260808 06:57:21.199887 29390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:21.200141 29390 log.cc:824] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:21.200735 29390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a: No bootstrap required, opened a new log
I20260808 06:57:21.202237 29390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffac90d0dfe424e974b3ee0cd94a35a" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40811 } }
I20260808 06:57:21.202368 29390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:21.202422 29390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ffac90d0dfe424e974b3ee0cd94a35a, State: Initialized, Role: FOLLOWER
I20260808 06:57:21.202548 29390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffac90d0dfe424e974b3ee0cd94a35a" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40811 } }
I20260808 06:57:21.202620 29390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:21.202652 29390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:21.202700 29390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:21.203188 29390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffac90d0dfe424e974b3ee0cd94a35a" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40811 } }
I20260808 06:57:21.203274 29390 leader_election.cc:302] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ffac90d0dfe424e974b3ee0cd94a35a; no voters:
I20260808 06:57:21.203441 29390 leader_election.cc:288] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:21.203490 29397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:21.203644 29397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [term 1 LEADER]: Becoming Leader. State: Replica: 6ffac90d0dfe424e974b3ee0cd94a35a, State: Running, Role: LEADER
I20260808 06:57:21.203709 29390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:21.203753 29397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffac90d0dfe424e974b3ee0cd94a35a" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40811 } }
I20260808 06:57:21.204082 29398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ffac90d0dfe424e974b3ee0cd94a35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffac90d0dfe424e974b3ee0cd94a35a" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40811 } } }
I20260808 06:57:21.204161 29398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:21.204257 29400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ffac90d0dfe424e974b3ee0cd94a35a. Latest consensus state: current_term: 1 leader_uuid: "6ffac90d0dfe424e974b3ee0cd94a35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffac90d0dfe424e974b3ee0cd94a35a" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40811 } } }
I20260808 06:57:21.204308 29400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:21.204432 29408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:21.204897 29408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:21.206089 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 29289
I20260808 06:57:21.206187 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:21.206365 29408 catalog_manager.cc:1347] Generated new cluster ID: acaa63404e164f21a82ecb79e203f9f8
I20260808 06:57:21.206383 21617 external_mini_cluster.cc:1428] Setting key 49a2d2a229176555e294926a9b9b9fc3
I20260808 06:57:21.206429 29408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:21.211954 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:21.207769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.21.28.126:40811)
0808 06:57:21.207961 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:21.207968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:21.207998 (+ 30us) server_negotiation.cc:407] Connection header received
0808 06:57:21.208092 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:21.208097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:21.208157 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:21.208275 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:21.208488 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.209361 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.210118 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.210514 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.210617 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:21.211205 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:21.211209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:21.211209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:21.211372 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:21.211406 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:21.211425 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:21.211483 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:21.211565 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 06:57:21.211640 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260808 06:57:21.212785 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:57:21.220978 29408 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:21.221431 29408 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:21.230221 29408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ffac90d0dfe424e974b3ee0cd94a35a: Generated new TSK 0
I20260808 06:57:21.230353 29408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:57:21.245172 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:21.241730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.21.28.126:40811)
0808 06:57:21.241773 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:21.241775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:21.241832 (+ 57us) server_negotiation.cc:407] Connection header received
0808 06:57:21.241883 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:21.241884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:21.241908 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:21.241974 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0808 06:57:21.242067 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.242847 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.243536 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.243888 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.243951 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:21.244068 (+ 117us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 06:57:21.244979 (+ 911us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 06:57:21.245104 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 06:57:21.245230 29425 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 06:57:21.247010 29434 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:21.241634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40811 (local address 127.0.0.1:41668)
0808 06:57:21.241765 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:21.241775 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:21.241826 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:21.241971 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:21.241972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:21.241973 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0808 06:57:21.242015 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:21.242017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.242910 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.242911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:21.243474 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:21.243476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.243949 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.243950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:21.243988 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:21.243992 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 06:57:21.245068 (+ 1076us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 06:57:21.246914 (+ 1846us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.28.126:40811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":1740,"thread_start_us":78,"threads_started":1}
I20260808 06:57:21.247577 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 29289
E20260808 06:57:21.258163 29435 webserver.cc:533] Webserver: sslize failed:
E20260808 06:57:21.258232 29435 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T06:57:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T06:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:21Z Disabled control of system clock
I20260808 06:57:21.296093 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:41121
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:41121
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/testchainca.cert with env {}
W20260808 06:57:21.382318 29439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:21.382512 29439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:21.382534 29439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:21.383616 29439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:21.383648 29439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:21.383658 29439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:21.383685 29439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:21.383701 29439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 06:57:21.383723 29439 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 06:57:21.383742 29439 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 06:57:21.385203 29439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:41121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:41121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:21.385387 29439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:21.385540 29439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:21.390029 29445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:21.390182 29439 server_base.cc:1034] running on GCE node
W20260808 06:57:21.390015 29447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:21.390017 29449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:21.390496 29439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:21.390759 29439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:21.391894 29439 hybrid_clock.cc:648] HybridClock initialized: now 1786172241391875 us; error 30 us; skew 500 ppm
I20260808 06:57:21.392987 29439 webserver.cc:460] Webserver started at http://127.21.28.126:44701/ using document root <none> and password file <none>
I20260808 06:57:21.393183 29439 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:21.393229 29439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:21.393350 29439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:21.394296 29439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "3a10b00f4ada41a6a7591a5b7850d75f"
format_stamp: "Formatted at 2026-08-08 06:57:21 on dist-test-slave-nj21"
server_key: "e10f71e23d36100ba7fc5964c23f87a9"
server_key_iv: "a82fbfe36ee0bf57c369a6b1d655ec4b"
server_key_version: "encryptionkey@0"
I20260808 06:57:21.394613 29439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "3a10b00f4ada41a6a7591a5b7850d75f"
format_stamp: "Formatted at 2026-08-08 06:57:21 on dist-test-slave-nj21"
server_key: "e10f71e23d36100ba7fc5964c23f87a9"
server_key_iv: "a82fbfe36ee0bf57c369a6b1d655ec4b"
server_key_version: "encryptionkey@0"
I20260808 06:57:21.395793 29439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:21.396451 29458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:21.396600 29439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:21.396682 29439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "3a10b00f4ada41a6a7591a5b7850d75f"
format_stamp: "Formatted at 2026-08-08 06:57:21 on dist-test-slave-nj21"
server_key: "e10f71e23d36100ba7fc5964c23f87a9"
server_key_iv: "a82fbfe36ee0bf57c369a6b1d655ec4b"
server_key_version: "encryptionkey@0"
I20260808 06:57:21.396744 29439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:21.414039 29439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:21.414299 29439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:21.414420 29439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:21.418056 29439 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:41121
I20260808 06:57:21.418088 29545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:41121 every 8 connection(s)
I20260808 06:57:21.418378 29439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:21.419266 29547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:21.420009 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 29439
I20260808 06:57:21.420104 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:21.420271 21617 external_mini_cluster.cc:1428] Setting key cb255bc8171c3a218dd6734ee815ad83
I20260808 06:57:21.422284 29547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f: Bootstrap starting.
I20260808 06:57:21.422967 29547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:21.423252 29547 log.cc:824] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:21.423926 29547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f: No bootstrap required, opened a new log
I20260808 06:57:21.425837 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:21.420738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.21.28.126:41121)
0808 06:57:21.420944 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:21.420953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:21.420986 (+ 33us) server_negotiation.cc:407] Connection header received
0808 06:57:21.421461 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:21.421470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:21.421547 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:21.421688 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:21.421832 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.423102 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.423879 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.424456 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.424569 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:21.425159 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:21.425163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:21.425164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:21.425364 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:21.425411 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:21.425425 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:21.425484 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:21.425540 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 06:57:21.425617 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260808 06:57:21.426221 29547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10b00f4ada41a6a7591a5b7850d75f" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41121 } }
I20260808 06:57:21.426798 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:57:21.426383 29547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:21.427564 29547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a10b00f4ada41a6a7591a5b7850d75f, State: Initialized, Role: FOLLOWER
I20260808 06:57:21.427722 29547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10b00f4ada41a6a7591a5b7850d75f" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41121 } }
I20260808 06:57:21.427804 29547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:21.427834 29547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:21.427878 29547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:21.428381 29547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10b00f4ada41a6a7591a5b7850d75f" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41121 } }
I20260808 06:57:21.428475 29547 leader_election.cc:302] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a10b00f4ada41a6a7591a5b7850d75f; no voters:
I20260808 06:57:21.428620 29547 leader_election.cc:288] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:21.428660 29554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:21.428788 29554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [term 1 LEADER]: Becoming Leader. State: Replica: 3a10b00f4ada41a6a7591a5b7850d75f, State: Running, Role: LEADER
I20260808 06:57:21.428850 29547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:21.428870 29554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10b00f4ada41a6a7591a5b7850d75f" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41121 } }
I20260808 06:57:21.429199 29557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a10b00f4ada41a6a7591a5b7850d75f. Latest consensus state: current_term: 1 leader_uuid: "3a10b00f4ada41a6a7591a5b7850d75f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10b00f4ada41a6a7591a5b7850d75f" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41121 } } }
I20260808 06:57:21.429296 29557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:21.429195 29555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a10b00f4ada41a6a7591a5b7850d75f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10b00f4ada41a6a7591a5b7850d75f" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41121 } } }
I20260808 06:57:21.429411 29555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:21.429575 29564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:21.430090 29564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:21.431634 29564 catalog_manager.cc:1347] Generated new cluster ID: 0afede7cd115460b8806f64dd0c6a14c
I20260808 06:57:21.431707 29564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:21.441713 29564 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:21.442418 29564 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:21.448390 29564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a10b00f4ada41a6a7591a5b7850d75f: Generated new TSK 0
I20260808 06:57:21.448540 29564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:57:21.610199 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:21.453702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.21.28.126:41121)
0808 06:57:21.453742 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:21.453743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:21.453798 (+ 55us) server_negotiation.cc:407] Connection header received
0808 06:57:21.453878 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:21.453878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:21.453887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:21.453943 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0808 06:57:21.454055 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.454734 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.455841 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.456032 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.456078 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:21.456231 (+ 153us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 06:57:21.609978 (+153747us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 06:57:21.610135 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49238: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 06:57:21.610268 29551 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49238: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 06:57:21.610368 29589 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:21.453618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:41121 (local address 127.0.0.1:49238)
0808 06:57:21.453737 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:21.453747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:21.453816 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:21.453944 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:21.453945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:21.453947 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 06:57:21.453998 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:21.454000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.454814 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.454815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:21.455775 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:21.455778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:21.456091 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:21.456092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:21.456152 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:21.456157 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 06:57:21.610132 (+153975us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 06:57:21.610257 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.28.126:41121: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260808 06:57:21.610669 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 29439
2026-08-08T06:57:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (346 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:21 dist-test-slave-nj21 krb5kdc[29602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:21 dist-test-slave-nj21 krb5kdc[29602](info): set up 2 sockets
Aug 08 06:57:21 dist-test-slave-nj21 krb5kdc[29602](info): commencing operation
krb5kdc: starting...
W20260808 06:57:22.928974 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:23.040309 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:41419
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:41419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:23.119834 29616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:23.120035 29616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:23.120065 29616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:23.121212 29616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:23.121251 29616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:23.121270 29616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:23.121286 29616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:23.121302 29616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:23.122740 29616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:41419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:41419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:23.122972 29616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:23.123174 29616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:23.127857 29624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:23.127869 29629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:23.128032 29616 server_base.cc:1034] running on GCE node
W20260808 06:57:23.128224 29626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:23.128486 29616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:23.128720 29616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:23.129859 29616 hybrid_clock.cc:648] HybridClock initialized: now 1786172243129835 us; error 38 us; skew 500 ppm
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.132200 29616 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:23.132588 29616 webserver.cc:460] Webserver started at http://127.21.28.126:34245/ using document root <none> and password file <none>
I20260808 06:57:23.132758 29616 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:23.132795 29616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:23.132884 29616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:23.133890 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "75cee06fcd764d1395545a008249f2ce"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "9b898c5976b7ba753014401a1bbc832c"
server_key_iv: "9c34df5ff20b99cc4681c4ad632fe43e"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.134177 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "75cee06fcd764d1395545a008249f2ce"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "9b898c5976b7ba753014401a1bbc832c"
server_key_iv: "9c34df5ff20b99cc4681c4ad632fe43e"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.135373 29616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:23.136034 29638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.136174 29616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.136264 29616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "75cee06fcd764d1395545a008249f2ce"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "9b898c5976b7ba753014401a1bbc832c"
server_key_iv: "9c34df5ff20b99cc4681c4ad632fe43e"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.136319 29616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:23.143429 29616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:23.145033 29616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:23.145135 29616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:23.148700 29616 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:41419
I20260808 06:57:23.148743 29721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:41419 every 8 connection(s)
I20260808 06:57:23.149027 29616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:23.149827 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:23.152016 29722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce: Bootstrap starting.
I20260808 06:57:23.152523 29722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:23.152755 29722 log.cc:824] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:23.153321 29722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce: No bootstrap required, opened a new log
I20260808 06:57:23.154186 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 29616
I20260808 06:57:23.154274 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:23.154449 21617 external_mini_cluster.cc:1428] Setting key b1a3a6735c9d905f1a3e6a303196a906
I20260808 06:57:23.155058 29722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cee06fcd764d1395545a008249f2ce" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41419 } }
I20260808 06:57:23.155222 29722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:23.155263 29722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cee06fcd764d1395545a008249f2ce, State: Initialized, Role: FOLLOWER
I20260808 06:57:23.155381 29722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cee06fcd764d1395545a008249f2ce" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41419 } }
I20260808 06:57:23.155459 29722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:23.155505 29722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:23.155550 29722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:23.156172 29722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cee06fcd764d1395545a008249f2ce" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41419 } }
I20260808 06:57:23.156275 29722 leader_election.cc:302] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75cee06fcd764d1395545a008249f2ce; no voters:
I20260808 06:57:23.156414 29722 leader_election.cc:288] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:23.156466 29727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:23.156610 29727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [term 1 LEADER]: Becoming Leader. State: Replica: 75cee06fcd764d1395545a008249f2ce, State: Running, Role: LEADER
I20260808 06:57:23.156651 29722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:23.156704 29727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cee06fcd764d1395545a008249f2ce" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41419 } }
I20260808 06:57:23.157019 29727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75cee06fcd764d1395545a008249f2ce. Latest consensus state: current_term: 1 leader_uuid: "75cee06fcd764d1395545a008249f2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cee06fcd764d1395545a008249f2ce" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41419 } } }
I20260808 06:57:23.157091 29727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:23.157235 29728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75cee06fcd764d1395545a008249f2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cee06fcd764d1395545a008249f2ce" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 41419 } } }
I20260808 06:57:23.157294 29728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce [sys.catalog]: This master's current role is: LEADER
W20260808 06:57:23.158317 29744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:57:23.158375 29744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:57:23.158435 29735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:23.158602 29735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:23.159989 29735 catalog_manager.cc:1347] Generated new cluster ID: d05a016b050741abba6b86e0c3f8878b
I20260808 06:57:23.160043 29735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:23.164740 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.154900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.21.28.126:41419)
0808 06:57:23.155094 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.155102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.155132 (+ 30us) server_negotiation.cc:407] Connection header received
0808 06:57:23.155234 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.155241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.155295 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.155452 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.156107 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.157465 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.158479 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.158938 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.159030 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.161399 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:23.161403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:23.161403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:23.161421 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:23.162919 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.163221 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.163224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.163224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.163320 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.163487 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.163488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.163488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.163774 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:23.164443 (+ 669us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:23.164499 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 06:57:23.164602 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260808 06:57:23.173131 29735 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:23.173786 29735 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:23.178113 29735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75cee06fcd764d1395545a008249f2ce: Generated new TSK 0
I20260808 06:57:23.178261 29735 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:23.201551 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41419
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:23.280682 29754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:23.280874 29754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:23.280895 29754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:23.282003 29754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:23.282042 29754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:23.282071 29754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:23.283478 29754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:23.283712 29754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:23.283922 29754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:23.288507 29764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:23.288574 29760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:23.288635 29754 server_base.cc:1034] running on GCE node
W20260808 06:57:23.288507 29761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:23.288877 29754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:23.289064 29754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:23.290210 29754 hybrid_clock.cc:648] HybridClock initialized: now 1786172243290195 us; error 27 us; skew 500 ppm
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.292660 29754 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:23.293053 29754 webserver.cc:460] Webserver started at http://127.21.28.65:34417/ using document root <none> and password file <none>
I20260808 06:57:23.293221 29754 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:23.293258 29754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:23.293354 29754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:23.294272 29754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "b6e7e386580d4a9990472df0e1355166"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "b1fe4ccfbcc1de9c629f8d606482d2ac"
server_key_iv: "1fb6e58c4540bb8246730982492cb447"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.294565 29754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "b6e7e386580d4a9990472df0e1355166"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "b1fe4ccfbcc1de9c629f8d606482d2ac"
server_key_iv: "1fb6e58c4540bb8246730982492cb447"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.295758 29754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:23.296366 29775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.296511 29754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:23.296593 29754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "b6e7e386580d4a9990472df0e1355166"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "b1fe4ccfbcc1de9c629f8d606482d2ac"
server_key_iv: "1fb6e58c4540bb8246730982492cb447"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.296650 29754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:23.303892 29754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:23.305478 29754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:23.305583 29754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:23.305752 29754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:23.306046 29754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:23.306082 29754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.306102 29754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:23.306118 29754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.311661 29754 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:37577
I20260808 06:57:23.311749 29942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:37577 every 8 connection(s)
I20260808 06:57:23.312032 29754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:23.315694 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 29754
I20260808 06:57:23.315783 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:23.315954 21617 external_mini_cluster.cc:1428] Setting key 9bd466e596ebf4b648b5a74a4ea8f886
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:23.320818 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.312697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:55373 (local address 127.21.28.126:41419)
0808 06:57:23.312746 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.312748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.312944 (+ 196us) server_negotiation.cc:407] Connection header received
0808 06:57:23.313800 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.313801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.313815 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.313862 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.314387 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.314917 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.315875 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.316122 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.316193 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.318358 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:23.318360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:23.318361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:23.318370 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:23.319603 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.319975 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.319976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.319977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.320029 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.320239 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.320240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.320240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.320346 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:23.320402 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:23.320650 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 06:57:23.320729 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:23.320966 29946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.312670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:41419 (local address 127.21.28.65:55373)
0808 06:57:23.312827 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:23.312861 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:23.313722 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:23.313918 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:23.313921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:23.314153 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:23.314327 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:23.314331 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.314979 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.314979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:23.315801 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:23.315804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.316231 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.316232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:23.316358 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.316835 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:23.316843 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:23.318273 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:23.319691 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:23.319698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:23.319700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:23.319910 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:23.320082 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:23.320083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:23.320083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:23.320184 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:23.320408 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:23.320414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:23.320537 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 06:57:23.320620 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:57:23.320763 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260808 06:57:23.321439 29943 heartbeater.cc:347] Connected to a master server at 127.21.28.126:41419
I20260808 06:57:23.321542 29943 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:23.321727 29943 heartbeater.cc:511] Master 127.21.28.126:41419 requested a full tablet report, sending...
I20260808 06:57:23.322206 29669 ts_manager.cc:194] Registered new tserver with Master: b6e7e386580d4a9990472df0e1355166 (127.21.28.65:37577)
I20260808 06:57:23.322783 29669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:55373
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:23.340448 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41419
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:23.419382 29949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:23.419566 29949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:23.419588 29949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:23.420696 29949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:23.420737 29949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:23.420765 29949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:23.422178 29949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:23.422376 29949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:23.422572 29949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:23.427157 29961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:23.427209 29957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:23.427168 29964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:23.427197 29949 server_base.cc:1034] running on GCE node
I20260808 06:57:23.427588 29949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:23.427811 29949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:23.428952 29949 hybrid_clock.cc:648] HybridClock initialized: now 1786172243428940 us; error 27 us; skew 500 ppm
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.431264 29949 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:23.431643 29949 webserver.cc:460] Webserver started at http://127.21.28.66:38945/ using document root <none> and password file <none>
I20260808 06:57:23.431852 29949 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:23.431895 29949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:23.432029 29949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:23.433050 29949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "db390a10f825438dad1ae10ecc88bb33"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "f111dde7fc1a8df84e568eb276e5255f"
server_key_iv: "4306f963388bf8e2de7c7a1bc652a8d4"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.433364 29949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "db390a10f825438dad1ae10ecc88bb33"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "f111dde7fc1a8df84e568eb276e5255f"
server_key_iv: "4306f963388bf8e2de7c7a1bc652a8d4"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.434581 29949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:23.435192 29972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.435339 29949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:23.435421 29949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "db390a10f825438dad1ae10ecc88bb33"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "f111dde7fc1a8df84e568eb276e5255f"
server_key_iv: "4306f963388bf8e2de7c7a1bc652a8d4"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.435485 29949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:23.459990 29949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:23.462152 29949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:23.462289 29949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:23.462514 29949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:23.462810 29949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:23.462842 29949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.462870 29949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:23.462894 29949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.468516 29949 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:33249
I20260808 06:57:23.468555 30146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:33249 every 8 connection(s)
I20260808 06:57:23.468847 29949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:23.474882 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 29949
I20260808 06:57:23.474967 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:23.475175 21617 external_mini_cluster.cc:1428] Setting key db3bf7cdd630a7d2647ca4985ccf0f75
I20260808 06:57:23.477321 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.469507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:51827 (local address 127.21.28.126:41419)
0808 06:57:23.469560 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.469562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.469756 (+ 194us) server_negotiation.cc:407] Connection header received
0808 06:57:23.470649 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.470651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.470663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.470715 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.471288 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.471818 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.472761 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.472958 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.473012 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.474843 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:23.474845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:23.474845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:23.474852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:23.476138 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.476480 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.476481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.476482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.476538 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.476739 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.476740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.476741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.476848 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:23.476913 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:23.477150 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 06:57:23.477230 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 06:57:23.477591 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.469481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:41419 (local address 127.21.28.66:51827)
0808 06:57:23.469643 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:23.469676 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:23.470551 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:23.470779 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:23.470783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:23.471028 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:23.471206 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:23.471210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.471900 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.471901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:23.472685 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:23.472688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.473024 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.473025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:23.473146 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.473513 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:23.473520 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:23.474759 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:23.476205 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:23.476211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:23.476213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:23.476418 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:23.476588 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:23.476589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:23.476589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:23.476686 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:23.476901 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:23.476906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:23.477039 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 06:57:23.477122 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:57:23.477265 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260808 06:57:23.477972 30147 heartbeater.cc:347] Connected to a master server at 127.21.28.126:41419
I20260808 06:57:23.478070 30147 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:23.478245 30147 heartbeater.cc:511] Master 127.21.28.126:41419 requested a full tablet report, sending...
I20260808 06:57:23.478574 29669 ts_manager.cc:194] Registered new tserver with Master: db390a10f825438dad1ae10ecc88bb33 (127.21.28.66:33249)
I20260808 06:57:23.478925 29669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:51827
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:23.498596 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41419
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:23.577436 30153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:23.577625 30153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:23.577656 30153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:23.578765 30153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:23.578802 30153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:23.578837 30153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:23.580308 30153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:41419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:23.580566 30153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:23.580812 30153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:57:23.585523 30153 server_base.cc:1034] running on GCE node
W20260808 06:57:23.585438 30163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:23.585438 30162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:23.585480 30167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:23.585866 30153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:23.586071 30153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:23.587200 30153 hybrid_clock.cc:648] HybridClock initialized: now 1786172243587188 us; error 29 us; skew 500 ppm
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.589589 30153 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:23.589993 30153 webserver.cc:460] Webserver started at http://127.21.28.67:43979/ using document root <none> and password file <none>
I20260808 06:57:23.590201 30153 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:23.590246 30153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:23.590365 30153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:23.591321 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "169b9f59465947bf903cd595de1b862b"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "a91df19bcc0df4c5e2311e76f64f8ecd"
server_key_iv: "5c0c1b4d6501299c92ad577a53f014d0"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.591645 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "169b9f59465947bf903cd595de1b862b"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "a91df19bcc0df4c5e2311e76f64f8ecd"
server_key_iv: "5c0c1b4d6501299c92ad577a53f014d0"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.592845 30153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:23.593467 30178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.593612 30153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:23.593683 30153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "169b9f59465947bf903cd595de1b862b"
format_stamp: "Formatted at 2026-08-08 06:57:23 on dist-test-slave-nj21"
server_key: "a91df19bcc0df4c5e2311e76f64f8ecd"
server_key_iv: "5c0c1b4d6501299c92ad577a53f014d0"
server_key_version: "encryptionkey@0"
I20260808 06:57:23.593747 30153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:23.603061 30153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:23.604961 30153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:23.605084 30153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:23.605289 30153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:23.605571 30153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:23.605603 30153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.605630 30153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:23.605654 30153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:23.611357 30153 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:33119
I20260808 06:57:23.611392 30351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:33119 every 8 connection(s)
I20260808 06:57:23.611721 30153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:23.612529 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 30153
I20260808 06:57:23.612640 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:23.612838 21617 external_mini_cluster.cc:1428] Setting key 8337dbb1e627deefc81b345cdc65a4e7
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:23.619752 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.612312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:41947 (local address 127.21.28.126:41419)
0808 06:57:23.612386 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.612388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.612628 (+ 240us) server_negotiation.cc:407] Connection header received
0808 06:57:23.613699 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.613700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.613711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.613751 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.614278 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.614754 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.615640 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.615866 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.615914 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.617744 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:23.617745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:23.617745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:23.617751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:23.618753 (+ 1002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.619067 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.619067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.619067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.619110 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.619271 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.619272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.619272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.619359 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:23.619407 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:23.619599 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 06:57:23.619651 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 06:57:23.619968 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.612289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:41419 (local address 127.21.28.67:41947)
0808 06:57:23.612481 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:23.612528 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:23.613617 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:23.613809 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:23.613812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:23.614043 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:23.614219 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:23.614223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.614815 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.614816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:23.615579 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:23.615581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.615926 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.615926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:23.616023 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.616435 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:23.616443 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:23.617669 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:23.618811 (+ 1142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:23.618815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:23.618817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:23.619013 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:23.619151 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:23.619151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:23.619151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:23.619233 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:23.619397 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:23.619400 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:23.619508 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 06:57:23.619579 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 06:57:23.619762 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260808 06:57:23.620357 30352 heartbeater.cc:347] Connected to a master server at 127.21.28.126:41419
I20260808 06:57:23.620430 30352 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:23.620581 30352 heartbeater.cc:511] Master 127.21.28.126:41419 requested a full tablet report, sending...
I20260808 06:57:23.620915 29669 ts_manager.cc:194] Registered new tserver with Master: 169b9f59465947bf903cd595de1b862b (127.21.28.67:33119)
I20260808 06:57:23.621254 29669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:41947
I20260808 06:57:23.624426 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/test-admin.keytab.
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.638931 21617 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 06:57:23.640543 30363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:23.639758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.21.28.65:37577)
0808 06:57:23.639941 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.639945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.639976 (+ 31us) server_negotiation.cc:407] Connection header received
0808 06:57:23.640029 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.640031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.640045 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.640115 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.640484 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37288: BlockingRecv error: recv got EOF from 127.0.0.1:37288 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
W20260808 06:57:23.640645 21617 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5cc' of type FILE
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.641556 21617 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.642319 21617 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:23.649259 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.642787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.21.28.65:37577)
0808 06:57:23.642835 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.642837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.642866 (+ 29us) server_negotiation.cc:407] Connection header received
0808 06:57:23.642934 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.642935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.642947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.642990 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.643338 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.644091 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.644776 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.645084 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.645142 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.646909 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:23.646918 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:23.646918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:23.646926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:23.648271 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.648569 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.648571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.648571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.648622 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.648780 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.648781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.648781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.648902 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:23.649021 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:23.649095 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 06:57:23.649165 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260808 06:57:23.650696 30363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:23.650300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.21.28.65:37577)
0808 06:57:23.650329 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.650331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.650361 (+ 30us) server_negotiation.cc:407] Connection header received
0808 06:57:23.650401 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.650402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.650415 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.650457 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.650657 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37318: BlockingRecv error: recv got EOF from 127.0.0.1:37318 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260808 06:57:23.650810 21617 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5cc' of type FILE
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.651618 21617 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.652345 21617 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:23.658371 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.652772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.21.28.65:37577)
0808 06:57:23.652815 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.652816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.652824 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:23.652892 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.652892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.652901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.652940 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.653257 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.653712 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.654338 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.654532 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.654583 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.656296 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:23.656297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:23.656297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:23.656303 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:23.657465 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.657747 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.657747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.657748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.657793 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.657960 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.657960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.657960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.658036 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:23.658108 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:23.658232 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 06:57:23.658304 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260808 06:57:23.659593 30363 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:23.659215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.21.28.65:37577)
0808 06:57:23.659243 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.659245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.659280 (+ 35us) server_negotiation.cc:407] Connection header received
0808 06:57:23.659320 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.659320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.659326 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.659369 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.659557 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37324: BlockingRecv error: recv got EOF from 127.0.0.1:37324 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260808 06:57:23.659808 21617 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786172218768454-21617-0/krb5kdc/krb5cc' of type FILE
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.660605 21617 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:23.661329 21617 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[29602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.65@KRBTEST.COM
I20260808 06:57:23.666880 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:23.661727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.21.28.65:37577)
0808 06:57:23.661779 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:23.661781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:23.661790 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:23.661840 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:23.661841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:23.661847 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:23.661886 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:23.662167 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.662590 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.663188 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:23.663349 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:23.663388 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:23.665072 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:23.665073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:23.665073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:23.665079 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:23.666169 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.666421 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.666421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.666421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.666462 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:23.666575 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:23.666575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:23.666576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:23.666642 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:23.666696 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:23.666764 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 06:57:23.666811 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 06:57:23.667410 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 29754
I20260808 06:57:23.680933 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 29949
I20260808 06:57:23.687969 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 30153
I20260808 06:57:23.696256 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 29616
2026-08-08T06:57:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1907 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[30385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[30385](info): set up 2 sockets
Aug 08 06:57:23 dist-test-slave-nj21 krb5kdc[30385](info): commencing operation
krb5kdc: starting...
W20260808 06:57:24.827809 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:24 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.126.keytab.
Entry for principal oryx/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.126.keytab.
I20260808 06:57:24.939888 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:37545
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.126.keytab
--principal=oryx/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:25.018261 30401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:25.018442 30401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:25.018463 30401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:25.019563 30401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:25.019604 30401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:25.019618 30401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:25.019639 30401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:25.019654 30401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:25.021107 30401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.126.keytab
--principal=oryx/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:25.021313 30401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:25.021513 30401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:25.025907 30412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:25.026021 30401 server_base.cc:1034] running on GCE node
W20260808 06:57:25.025928 30408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:25.025988 30409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:25.026337 30401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:25.026603 30401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:25.027722 30401 hybrid_clock.cc:648] HybridClock initialized: now 1786172245027704 us; error 28 us; skew 500 ppm
Aug 08 06:57:25 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172245, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:25.030066 30401 init.cc:377] Logged in from keytab as oryx/127.21.28.126@KRBTEST.COM (short username oryx)
I20260808 06:57:25.030440 30401 webserver.cc:460] Webserver started at http://127.21.28.126:35831/ using document root <none> and password file <none>
I20260808 06:57:25.030643 30401 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:25.030689 30401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:25.030807 30401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:25.031809 30401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "e1fb889f0bcc524dae86d25cd5313874"
server_key_iv: "4f05105d930dba90b0599fd5b8a28614"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.032121 30401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "e1fb889f0bcc524dae86d25cd5313874"
server_key_iv: "4f05105d930dba90b0599fd5b8a28614"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.033291 30401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:25.033947 30421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:25.034102 30401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:25.034190 30401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "e1fb889f0bcc524dae86d25cd5313874"
server_key_iv: "4f05105d930dba90b0599fd5b8a28614"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.034248 30401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:25.058683 30401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:25.060616 30401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:25.060750 30401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:25.064473 30401 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:37545
I20260808 06:57:25.064512 30497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:37545 every 8 connection(s)
I20260808 06:57:25.064823 30401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:25.065737 30500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:25.068056 30500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: Bootstrap starting.
I20260808 06:57:25.068586 30500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:25.068843 30500 log.cc:824] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:25.069478 30500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: No bootstrap required, opened a new log
I20260808 06:57:25.071004 30500 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } }
I20260808 06:57:25.071134 30500 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:25.071167 30500 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fa7e28f9544dd1a57bfbf8bb65d062, State: Initialized, Role: FOLLOWER
I20260808 06:57:25.071306 30500 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } }
I20260808 06:57:25.071382 30500 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:25.071419 30500 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:25.071465 30500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:25.072255 30500 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } }
I20260808 06:57:25.072346 30500 leader_election.cc:302] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1fa7e28f9544dd1a57bfbf8bb65d062; no voters:
I20260808 06:57:25.072489 30500 leader_election.cc:288] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:25.072521 30504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:25.072638 30504 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 1 LEADER]: Becoming Leader. State: Replica: e1fa7e28f9544dd1a57bfbf8bb65d062, State: Running, Role: LEADER
I20260808 06:57:25.072727 30500 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:25.072777 30504 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } }
I20260808 06:57:25.073122 30506 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1fa7e28f9544dd1a57bfbf8bb65d062. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } }
I20260808 06:57:25.073112 30505 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } }
I20260808 06:57:25.073307 30505 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:25.073231 30506 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:25.073678 30515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:25.074167 30515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:25.074205 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 30401
I20260808 06:57:25.074307 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:25.074520 21617 external_mini_cluster.cc:1428] Setting key cbd1a2b521e6786784acf876ff1b125e
I20260808 06:57:25.075738 30515 catalog_manager.cc:1347] Generated new cluster ID: 28a6f53c4a714db3aa2fb5b40ce56110
I20260808 06:57:25.075789 30515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:57:25 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:25.084014 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:25.074915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.21.28.126:37545)
0808 06:57:25.075081 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:25.075090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:25.075269 (+ 179us) server_negotiation.cc:407] Connection header received
0808 06:57:25.075500 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:25.075508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:25.075564 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:25.075733 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:25.076428 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.077323 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.078030 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.078462 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.078561 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:25.080727 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:25.080731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:25.080732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:25.080743 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:25.082614 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:25.082919 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:25.082921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:25.082922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:25.083013 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:25.083184 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:25.083185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:25.083185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:25.083417 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:25.083556 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:25.083653 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 06:57:25.083807 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260808 06:57:25.092638 30515 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:25.093326 30515 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:25.104187 30515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: Generated new TSK 0
I20260808 06:57:25.104398 30515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab.
Entry for principal oryx/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab.
I20260808 06:57:25.134716 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:37545
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab
--principal=oryx/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:25.215101 30539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:25.215287 30539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:25.215309 30539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:25.216446 30539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:25.216487 30539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:25.216514 30539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:25.217957 30539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab
--principal=oryx/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:25.218173 30539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:25.218389 30539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:57:25.223426 30539 server_base.cc:1034] running on GCE node
W20260808 06:57:25.223414 30549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:25.223388 30548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:25.223388 30552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:25.223881 30539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:25.224107 30539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:25.225255 30539 hybrid_clock.cc:648] HybridClock initialized: now 1786172245225239 us; error 28 us; skew 500 ppm
Aug 08 06:57:25 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172245, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:25.227581 30539 init.cc:377] Logged in from keytab as oryx/127.21.28.65@KRBTEST.COM (short username oryx)
I20260808 06:57:25.227984 30539 webserver.cc:460] Webserver started at http://127.21.28.65:35279/ using document root <none> and password file <none>
I20260808 06:57:25.228194 30539 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:25.228240 30539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:25.228368 30539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:25.229343 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "c6350088346744faacf9843062dd1e2b"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "d8b11085f38639b25854e6009cad0203"
server_key_iv: "b5de0819dd8e4109ff08799585315e65"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.229676 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "c6350088346744faacf9843062dd1e2b"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "d8b11085f38639b25854e6009cad0203"
server_key_iv: "b5de0819dd8e4109ff08799585315e65"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.230851 30539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:25.231478 30559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:25.231626 30539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:25.231734 30539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "c6350088346744faacf9843062dd1e2b"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "d8b11085f38639b25854e6009cad0203"
server_key_iv: "b5de0819dd8e4109ff08799585315e65"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.231794 30539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:25.241020 30539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:25.242543 30539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:25.242666 30539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:25.242861 30539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:25.243145 30539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:25.243178 30539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:25.243206 30539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:25.243232 30539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:25.248745 30539 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:38769
I20260808 06:57:25.248826 30717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:38769 every 8 connection(s)
I20260808 06:57:25.249056 30539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
Aug 08 06:57:25 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172245, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:25.257025 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:25.249718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:56629 (local address 127.21.28.126:37545)
0808 06:57:25.249775 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:25.249777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:25.249942 (+ 165us) server_negotiation.cc:407] Connection header received
0808 06:57:25.250389 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:25.250390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:25.250402 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:25.250454 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:25.251260 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.251831 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.252743 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.252951 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.253000 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:25.254855 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:25.254857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:25.254857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:25.254862 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:25.256034 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:25.256361 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:25.256362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:25.256362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:25.256411 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:25.256577 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:25.256577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:25.256577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:25.256668 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:25.256713 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:25.256899 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 06:57:25.256961 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:57:25.257169 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:25.249678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.65:56629)
0808 06:57:25.249830 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:25.249867 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:25.250317 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:25.250502 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:25.250505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:25.251022 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:25.251195 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:25.251200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.251893 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.251894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:25.252681 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:25.252683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.253011 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.253011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:25.253105 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:25.253541 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:25.253548 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:25.254776 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:25.256106 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:25.256111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:25.256113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:25.256309 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:25.256454 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:25.256455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:25.256455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:25.256537 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:25.256709 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:25.256711 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:25.256809 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 06:57:25.256879 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 06:57:25.256985 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260808 06:57:25.257649 30718 heartbeater.cc:347] Connected to a master server at 127.21.28.126:37545
I20260808 06:57:25.257730 30718 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:25.257886 30718 heartbeater.cc:511] Master 127.21.28.126:37545 requested a full tablet report, sending...
I20260808 06:57:25.258338 30447 ts_manager.cc:194] Registered new tserver with Master: c6350088346744faacf9843062dd1e2b (127.21.28.65:38769)
I20260808 06:57:25.258899 30447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:56629
I20260808 06:57:25.259276 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 30539
I20260808 06:57:25.259380 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:25.259581 21617 external_mini_cluster.cc:1428] Setting key f29b3aafd9ac1398727ecc2ab6872829
WARNING: no policy specified for oryx/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab.
Entry for principal oryx/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab.
W20260808 06:57:25.798548 21617 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.21.28.66: real 0.538s user 0.002s sys 0.000s
I20260808 06:57:25.798877 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:37545
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab
--principal=oryx/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:25.878113 30725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:25.878317 30725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:25.878340 30725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:25.879469 30725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:25.879509 30725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:25.879537 30725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:25.880975 30725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab
--principal=oryx/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:25.881181 30725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:25.881393 30725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:25.886054 30730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:25.886071 30732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:25.886068 30734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:25.886217 30725 server_base.cc:1034] running on GCE node
I20260808 06:57:25.886389 30725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:25.886618 30725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:25.887720 30725 hybrid_clock.cc:648] HybridClock initialized: now 1786172245887701 us; error 34 us; skew 500 ppm
Aug 08 06:57:25 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172245, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:25.889942 30725 init.cc:377] Logged in from keytab as oryx/127.21.28.66@KRBTEST.COM (short username oryx)
I20260808 06:57:25.890347 30725 webserver.cc:460] Webserver started at http://127.21.28.66:46875/ using document root <none> and password file <none>
I20260808 06:57:25.890568 30725 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:25.890616 30725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:25.890739 30725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:25.891700 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "e96629b8965148f98b194c92e1997d6a"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "7de86a47fd8d91f74ab3a0b2f645131e"
server_key_iv: "48b43c8973ccd0bac4369d3d59a5b4ba"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.892045 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "e96629b8965148f98b194c92e1997d6a"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "7de86a47fd8d91f74ab3a0b2f645131e"
server_key_iv: "48b43c8973ccd0bac4369d3d59a5b4ba"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.893225 30725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:25.893888 30746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:25.894040 30725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:25.894119 30725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "e96629b8965148f98b194c92e1997d6a"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "7de86a47fd8d91f74ab3a0b2f645131e"
server_key_iv: "48b43c8973ccd0bac4369d3d59a5b4ba"
server_key_version: "encryptionkey@0"
I20260808 06:57:25.894182 30725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:25.904644 30725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:25.906148 30725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:25.906268 30725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:25.906459 30725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:25.906733 30725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:25.906765 30725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:25.906790 30725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:25.906817 30725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:25.912665 30725 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:46533
I20260808 06:57:25.912722 30914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:46533 every 8 connection(s)
I20260808 06:57:25.912982 30725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:25.913476 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 30725
I20260808 06:57:25.913563 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:25.913733 21617 external_mini_cluster.cc:1428] Setting key 57c2406dd7a7bbdd60998a98dc6f3934
Aug 08 06:57:25 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172245, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:25.921970 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:25.913639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:35701 (local address 127.21.28.126:37545)
0808 06:57:25.913756 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:25.913759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:25.913953 (+ 194us) server_negotiation.cc:407] Connection header received
0808 06:57:25.914469 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:25.914470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:25.914482 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:25.914536 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:25.915414 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.915971 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.916909 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.917112 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.917166 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:25.919331 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:25.919332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:25.919333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:25.919339 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:25.920633 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:25.921061 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:25.921062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:25.921062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:25.921117 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:25.921359 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:25.921359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:25.921360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:25.921463 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:25.921506 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:25.921834 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 06:57:25.921899 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
WARNING: no policy specified for oryx/127.21.28.67@KRBTEST.COM; defaulting to no policy
I20260808 06:57:25.922187 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:25.913577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.66:35701)
0808 06:57:25.913841 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:25.913872 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:25.914381 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:25.914599 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:25.914603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:25.915143 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:25.915311 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:25.915315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.916050 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.916051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:25.916826 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:25.916828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:25.917179 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:25.917180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:25.917302 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:25.917757 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:25.917764 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:25.919243 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:25.920701 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:25.920708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:25.920710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:25.920994 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:25.921173 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:25.921173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:25.921174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:25.921302 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:25.921535 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:25.921541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:25.921695 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 06:57:25.921811 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 06:57:25.921972 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260808 06:57:25.922634 30915 heartbeater.cc:347] Connected to a master server at 127.21.28.126:37545
I20260808 06:57:25.922719 30915 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:25.922881 30915 heartbeater.cc:511] Master 127.21.28.126:37545 requested a full tablet report, sending...
I20260808 06:57:25.923224 30447 ts_manager.cc:194] Registered new tserver with Master: e96629b8965148f98b194c92e1997d6a (127.21.28.66:46533)
I20260808 06:57:25.923579 30447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:35701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab.
Entry for principal oryx/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab.
I20260808 06:57:25.938493 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:37545
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab
--principal=oryx/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:26.016372 30922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:26.016568 30922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:26.016600 30922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:26.017686 30922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:26.017724 30922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:26.017757 30922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:26.019167 30922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab
--principal=oryx/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:26.019388 30922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:26.019594 30922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:26.024091 30928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.024092 30933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.024202 30929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.024498 30922 server_base.cc:1034] running on GCE node
I20260808 06:57:26.024662 30922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:26.024833 30922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:26.025956 30922 hybrid_clock.cc:648] HybridClock initialized: now 1786172246025952 us; error 34 us; skew 500 ppm
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:26.031296 30922 init.cc:377] Logged in from keytab as oryx/127.21.28.67@KRBTEST.COM (short username oryx)
I20260808 06:57:26.031757 30922 webserver.cc:460] Webserver started at http://127.21.28.67:32859/ using document root <none> and password file <none>
I20260808 06:57:26.031962 30922 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.032008 30922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.032135 30922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:26.033100 30922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "4792a68b34154ac9872393a4934b71d1"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "96d8e6f180a20467a9537f7378974d78"
server_key_iv: "3b07708a334e3eba57bc851866daffe1"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.033432 30922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "4792a68b34154ac9872393a4934b71d1"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "96d8e6f180a20467a9537f7378974d78"
server_key_iv: "3b07708a334e3eba57bc851866daffe1"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.034607 30922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:26.035235 30946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.035414 30922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:26.035492 30922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "4792a68b34154ac9872393a4934b71d1"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "96d8e6f180a20467a9537f7378974d78"
server_key_iv: "3b07708a334e3eba57bc851866daffe1"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.035540 30922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.044569 30922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:26.046078 30922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:26.046176 30922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:26.046334 30922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:26.046576 30922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:26.046605 30922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.046623 30922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:26.046639 30922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.052202 30922 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:33083
I20260808 06:57:26.052278 31124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:33083 every 8 connection(s)
I20260808 06:57:26.052520 30922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:26.060286 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.053107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:33071 (local address 127.21.28.126:37545)
0808 06:57:26.053167 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.053168 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.053335 (+ 167us) server_negotiation.cc:407] Connection header received
0808 06:57:26.053769 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.053770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.053780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.053831 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.054638 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.055123 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.056059 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.056250 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.056296 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.058048 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.058049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.058050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.058056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.059276 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.059568 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.059568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.059569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.059613 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.059811 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.059811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.059811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.059915 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.059963 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.060148 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.060219 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 06:57:26.060405 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.053076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.67:33071)
0808 06:57:26.053223 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.053262 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.053698 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.053879 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.053882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.054398 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.054574 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.054577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.055181 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.055182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.055996 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.055998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.056303 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.056304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.056393 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.056744 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.056751 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.057974 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.059333 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.059336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.059338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.059519 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.059655 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.059656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.059656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.059765 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.059961 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.059963 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.060060 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.060131 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.060234 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260808 06:57:26.060791 31125 heartbeater.cc:347] Connected to a master server at 127.21.28.126:37545
I20260808 06:57:26.060869 31125 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:26.061026 31125 heartbeater.cc:511] Master 127.21.28.126:37545 requested a full tablet report, sending...
I20260808 06:57:26.061331 30447 ts_manager.cc:194] Registered new tserver with Master: 4792a68b34154ac9872393a4934b71d1 (127.21.28.67:33083)
I20260808 06:57:26.061669 30447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:33071
I20260808 06:57:26.062461 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 30922
I20260808 06:57:26.062554 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:26.062697 21617 external_mini_cluster.cc:1428] Setting key bcf2ccdbaa882e4d8379555952bd6752
I20260808 06:57:26.063390 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.28.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.125.keytab.
Entry for principal oryx/127.21.28.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.125.keytab.
I20260808 06:57:26.086193 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.125:40205
--webserver_interface=127.21.28.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.125.keytab
--principal=oryx/127.21.28.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:37545,127.21.28.125:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:26.169685 31134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:26.169902 31134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:26.169924 31134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:26.171053 31134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:26.171094 31134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:26.171108 31134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:26.171118 31134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:26.171142 31134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:26.172652 31134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:37545,127.21.28.125:40205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.125.keytab
--principal=oryx/127.21.28.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.125:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.28.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
W20260808 06:57:26.172768 31134 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.28.126:37545,127.21.28.125:40205'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 06:57:26.172912 31134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:26.173120 31134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:26.177651 31142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.177727 31146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.177662 31144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.177800 31134 server_base.cc:1034] running on GCE node
I20260808 06:57:26.178092 31134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:26.178318 31134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:26.179448 31134 hybrid_clock.cc:648] HybridClock initialized: now 1786172246179436 us; error 26 us; skew 500 ppm
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:26.181761 31134 init.cc:377] Logged in from keytab as oryx/127.21.28.125@KRBTEST.COM (short username oryx)
I20260808 06:57:26.182150 31134 webserver.cc:460] Webserver started at http://127.21.28.125:34111/ using document root <none> and password file <none>
I20260808 06:57:26.182358 31134 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.182404 31134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.182535 31134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:26.183537 31134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data/instance:
uuid: "cffb51c0521e4dec9fb886b67223f949"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "e0feea7ccc09abd771e591e9d1587b7a"
server_key_iv: "93a16ce22e1a7dc37243389f55c2453f"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.183884 31134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal/instance:
uuid: "cffb51c0521e4dec9fb886b67223f949"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "e0feea7ccc09abd771e591e9d1587b7a"
server_key_iv: "93a16ce22e1a7dc37243389f55c2453f"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.185086 31134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:26.185717 31154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.185858 31134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:26.185943 31134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
uuid: "cffb51c0521e4dec9fb886b67223f949"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "e0feea7ccc09abd771e591e9d1587b7a"
server_key_iv: "93a16ce22e1a7dc37243389f55c2453f"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.186003 31134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.202665 31134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:26.204294 31134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:26.204427 31134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:26.208251 31134 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.125:40205
I20260808 06:57:26.208289 31235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.125:40205 every 8 connection(s)
I20260808 06:57:26.208652 31134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
I20260808 06:57:26.209612 31237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:26.210520 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 31134
I20260808 06:57:26.211324 31237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.125@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.28.125@KRBTEST.COM
I20260808 06:57:26.219298 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.212010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.21.28.126:37545)
0808 06:57:26.212059 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.212061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.212206 (+ 145us) server_negotiation.cc:407] Connection header received
0808 06:57:26.212272 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.212273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.212285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.212346 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.212851 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.213352 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.214314 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.214565 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.214611 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.216718 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.216720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.216721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.216726 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.218014 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.218423 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.218424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.218424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.218478 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.218684 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.218684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.218685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.218825 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.218884 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.219127 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.219209 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 06:57:26.219446 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.211919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:45978)
0808 06:57:26.212093 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.212128 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.212210 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.212375 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.212377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.212622 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.212769 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.212773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.213436 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.213437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.214243 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.214245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.214625 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.214626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.214736 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.215214 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.215224 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.216643 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.218088 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.218093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.218095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.218359 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.218531 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.218532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.218532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.218630 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.218887 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.218892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.219015 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.219097 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.219236 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260808 06:57:26.219877 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.211296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.21.28.125:40205)
0808 06:57:26.211485 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.211495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.211526 (+ 31us) server_negotiation.cc:407] Connection header received
0808 06:57:26.212054 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.212063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.212125 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.212241 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.212852 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.213749 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.214440 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.214774 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.214834 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.217285 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.217291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.217292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.217298 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.218696 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.219012 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.219015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.219015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.219065 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.219229 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.219231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.219231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.219359 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.219499 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.219714 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.219783 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260808 06:57:26.220165 31237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.125@KRBTEST.COM for oryx/127.21.28.125@KRBTEST.COM
I20260808 06:57:26.225507 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.220373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.0.0.1:39648)
0808 06:57:26.220397 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.220406 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.220443 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.220548 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.220549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.220686 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.220721 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.220722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.221269 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.221270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.221733 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.221734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.222052 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.222053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.222107 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.222413 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.222418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.223408 (+ 990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.224697 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.224698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.224698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.224839 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.224991 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.224992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.224992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.225041 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.225227 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.225228 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.225261 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.225319 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.225407 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260808 06:57:26.225558 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.220401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.21.28.125:40205)
0808 06:57:26.220442 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.220443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.220451 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:26.220486 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.220487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.220494 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.220544 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.220779 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.221210 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.221788 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.222001 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.222050 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.223475 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.223476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.223476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.223482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.224631 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.224892 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.224893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.224893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.224943 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.225088 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.225089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.225089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.225180 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.225230 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.225344 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.225453 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":26}
I20260808 06:57:26.226625 31237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Bootstrap starting.
I20260808 06:57:26.227339 31237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:26.227569 31237 log.cc:824] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:26.228257 31237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: No bootstrap required, opened a new log
I20260808 06:57:26.229760 31237 raft_consensus.cc:348] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } }
I20260808 06:57:26.229875 31237 raft_consensus.cc:374] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:26.229897 31237 raft_consensus.cc:729] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffb51c0521e4dec9fb886b67223f949, State: Initialized, Role: FOLLOWER
I20260808 06:57:26.229988 31237 consensus_queue.cc:260] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } }
I20260808 06:57:26.230201 31245 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } } }
I20260808 06:57:26.230252 31237 sys_catalog.cc:564] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:26.230273 31245 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 06:57:26.231886 31263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:57:26.231972 31263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:57:26.233270 31134 master_runner.cc:428] Detected that this master cffb51c0521e4dec9fb886b67223f949 is joining an existing cluster
I20260808 06:57:26.233354 31134 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.28.125:40205
I20260808 06:57:26.240777 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.234463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.21.28.126:37545)
0808 06:57:26.234502 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.234504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.234631 (+ 127us) server_negotiation.cc:407] Connection header received
0808 06:57:26.234695 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.234696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.234707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.234756 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.235168 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.235938 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.236772 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.237025 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.237084 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.238009 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.238010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.238010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.238017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.239520 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.240012 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.240013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.240013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.240071 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.240260 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.240261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.240261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.240382 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.240424 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.240603 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.240680 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:26.240818 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.234372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.0.0.1:39656)
0808 06:57:26.234743 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.234751 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.234806 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.234919 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.234920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.235178 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.235246 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.235248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.236069 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.236070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.236938 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.236942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.237506 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.237507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.237554 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.237937 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.237941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.238276 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.239851 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.239852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.239852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.240033 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.240215 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.240215 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.240216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.240266 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.240504 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.240505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.240548 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.240611 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.240699 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260808 06:57:26.240890 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.234419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.21.28.125:40205)
0808 06:57:26.234454 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.234456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.234789 (+ 333us) server_negotiation.cc:407] Connection header received
0808 06:57:26.234852 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.234853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.234861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.234905 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.235319 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.236000 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.237011 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.237233 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.237283 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.238340 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.238341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.238341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.238347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.239788 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.240099 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.240100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.240100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.240155 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.240321 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.240321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.240322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.240442 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.240506 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.240640 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.240766 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":19}
I20260808 06:57:26.241641 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.234372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:45980)
0808 06:57:26.234556 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.234565 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.234634 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.234774 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.234775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.235016 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.235112 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.235114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.236017 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.236018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.236685 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.236688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.237105 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.237106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.237168 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.237526 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.237531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.237923 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.239605 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.239606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.239607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.239939 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.240139 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.240140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.240140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.240192 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.240460 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.240461 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.240497 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.241336 (+ 839us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.241432 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":39,"thread_start_us":90,"threads_started":1}
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172245, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.126@KRBTEST.COM for oryx/127.21.28.125@KRBTEST.COM
I20260808 06:57:26.250339 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.243516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.21.28.125:40205)
0808 06:57:26.243553 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.243555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.243739 (+ 184us) server_negotiation.cc:407] Connection header received
0808 06:57:26.243821 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.243822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.243829 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.243869 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.244314 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.244721 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.245755 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.245917 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.245955 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.247885 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.247886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.247886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.247892 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.249181 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.249618 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.249619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.249619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.249679 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.249871 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.249872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.249872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.249989 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.250052 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.250233 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.250287 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:26.250432 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.243451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.0.0.1:39658)
0808 06:57:26.243613 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.243659 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.243760 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.243877 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.243880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.244193 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.244252 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.244254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.244780 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.244781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.245680 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.245682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.245971 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.245972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.246050 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.246403 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.246408 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.247807 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.249263 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.249270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.249272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.249541 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.249745 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.249746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.249746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.249813 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.250054 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.250061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.250114 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.250209 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.250326 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260808 06:57:26.250978 30447 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.28.125:40205: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cffb51c0521e4dec9fb886b67223f949"
member_type: NON_VOTER
last_known_addr {
host: "127.21.28.125"
port: 40205
}
attrs {
promote: true
}
}
dest_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062"
cas_config_opid_index: -1
I20260808 06:57:26.251340 30447 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } }
I20260808 06:57:26.251977 31280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cffb51c0521e4dec9fb886b67223f949 (127.21.28.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } }
I20260808 06:57:26.252164 31194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:26.252955 31194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Refusing update from remote peer e1fa7e28f9544dd1a57bfbf8bb65d062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 06:57:26.253005 31280 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.28.125:40205
I20260808 06:57:26.253224 31279 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } }
I20260808 06:57:26.253319 31134 master.cc:537] Master@127.21.28.125:40205 shutting down...
I20260808 06:57:26.253427 31279 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:26.253568 31280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 06:57:26.254276 30427 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 06:57:26.254724 30427 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 -> Peer cffb51c0521e4dec9fb886b67223f949 (127.21.28.125:40205): Couldn't send request to peer cffb51c0521e4dec9fb886b67223f949. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 06:57:26.256276 31134 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:26.256356 31134 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:26.256414 31134 tablet_replica.cc:330] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: stopping tablet replica
I20260808 06:57:26.259865 30718 heartbeater.cc:503] Master 127.21.28.126:37545 was elected leader, sending a full tablet report...
I20260808 06:57:26.269591 31134 master.cc:552] Master@127.21.28.125:40205 shutdown complete.
I20260808 06:57:26.269644 31134 master_runner.cc:305] Clearing existing system tablet
I20260808 06:57:26.269783 31134 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.269832 31134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.270516 31134 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.002s
I20260808 06:57:26.271026 31288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.271142 31134 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.271200 31134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.271225 31134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
uuid: "cffb51c0521e4dec9fb886b67223f949"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "e0feea7ccc09abd771e591e9d1587b7a"
server_key_iv: "93a16ce22e1a7dc37243389f55c2453f"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.272156 31134 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:26.273756 31134 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 06:57:26.273826 31134 log.cc:1196] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 06:57:26.274021 31134 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Deleting consensus metadata
I20260808 06:57:26.274135 31134 master_runner.cc:315] Copying system tablet from 127.21.28.126:37545
I20260808 06:57:26.274746 31134 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: tablet copy: Beginning tablet copy session from remote peer at address 127.21.28.126:37545
I20260808 06:57:26.280704 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.275030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:45984)
0808 06:57:26.275142 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.275151 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.275196 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.275320 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.275320 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.275473 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.275530 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.275532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.276516 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.276517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.277052 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.277054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.277496 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.277497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.277542 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.277994 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.277997 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.278275 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.279852 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.279853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.279853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.280023 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.280209 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.280210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.280210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.280253 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.280479 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.280480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.280512 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.280554 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.280624 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260808 06:57:26.280756 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.275110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.21.28.126:37545)
0808 06:57:26.275169 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.275171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.275183 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:57:26.275241 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.275242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.275252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.275317 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.275588 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.276446 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.277124 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.277431 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.277493 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.278354 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.278356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.278356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.278364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.279783 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.280086 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.280087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.280088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.280151 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.280299 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.280299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.280299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.280427 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.280493 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.280579 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.280654 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 06:57:26.281049 30472 tablet_copy_service.cc:140] P e1fa7e28f9544dd1a57bfbf8bb65d062: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cffb51c0521e4dec9fb886b67223f949 ({username='oryx', principal='oryx/127.21.28.125@KRBTEST.COM'} at 127.0.0.1:45984)
I20260808 06:57:26.281181 30472 tablet_copy_service.cc:161] P e1fa7e28f9544dd1a57bfbf8bb65d062: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cffb51c0521e4dec9fb886b67223f949 at {username='oryx', principal='oryx/127.21.28.125@KRBTEST.COM'} at 127.0.0.1:45984: session id = cffb51c0521e4dec9fb886b67223f949-00000000000000000000000000000000
I20260808 06:57:26.282075 30472 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 06:57:26.282670 31134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 06:57:26.283921 31134 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: tablet copy: Starting download of 0 data blocks...
I20260808 06:57:26.284065 31134 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: tablet copy: Starting download of 1 WAL segments...
I20260808 06:57:26.284921 31134 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 06:57:26.285609 30472 tablet_copy_service.cc:342] P e1fa7e28f9544dd1a57bfbf8bb65d062: Request end of tablet copy session cffb51c0521e4dec9fb886b67223f949-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.28.125@KRBTEST.COM'} at 127.0.0.1:45984
I20260808 06:57:26.285666 30472 tablet_copy_service.cc:433] P e1fa7e28f9544dd1a57bfbf8bb65d062: ending tablet copy session cffb51c0521e4dec9fb886b67223f949-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cffb51c0521e4dec9fb886b67223f949
I20260808 06:57:26.288129 31134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:26.289237 31304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.289243 31301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.289335 31134 server_base.cc:1034] running on GCE node
W20260808 06:57:26.289494 31299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.289605 31134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:26.289783 31134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:26.290915 31134 hybrid_clock.cc:648] HybridClock initialized: now 1786172246290905 us; error 21 us; skew 500 ppm
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:26.292171 31134 init.cc:377] Logged in from keytab as oryx/127.21.28.125@KRBTEST.COM (short username oryx)
I20260808 06:57:26.292439 31134 webserver.cc:460] Webserver started at http://127.21.28.125:34683/ using document root <none> and password file <none>
I20260808 06:57:26.292549 31134 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.292583 31134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.293221 31134 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:26.293722 31313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.293836 31134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.293879 31134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
uuid: "cffb51c0521e4dec9fb886b67223f949"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "e0feea7ccc09abd771e591e9d1587b7a"
server_key_iv: "93a16ce22e1a7dc37243389f55c2453f"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.293931 31134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.300441 31134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:26.301684 31134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:26.304394 31134 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.125:40205
I20260808 06:57:26.304780 31384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.125:40205 every 8 connection(s)
I20260808 06:57:26.304778 31134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
I20260808 06:57:26.305682 31385 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:57:26.306180 31385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Bootstrap starting.
I20260808 06:57:26.309090 31385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:57:26.309329 31385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: Bootstrap complete.
I20260808 06:57:26.309549 31385 raft_consensus.cc:348] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } }
I20260808 06:57:26.309635 31385 raft_consensus.cc:729] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cffb51c0521e4dec9fb886b67223f949, State: Initialized, Role: LEARNER
I20260808 06:57:26.309680 31385 consensus_queue.cc:260] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } }
I20260808 06:57:26.309859 31385 sys_catalog.cc:564] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:26.309862 31390 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } }
I20260808 06:57:26.309976 31390 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: This master's current role is: LEARNER
I20260808 06:57:26.311508 31405 catalog_manager.cc:1259] Loaded cluster ID: 28a6f53c4a714db3aa2fb5b40ce56110
I20260808 06:57:26.311645 31405 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: loading cluster ID for follower catalog manager: success
I20260808 06:57:26.312554 31405 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: acquiring CA information for follower catalog manager: success
I20260808 06:57:26.312773 31405 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:57:26.328104 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.321409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.21.28.125:40205)
0808 06:57:26.321525 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.321527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.321537 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:26.321563 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.321564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.321575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.321652 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.322138 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.323196 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.323929 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.324257 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.324332 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.325428 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.325430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.325430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.325437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.327076 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.327377 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.327378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.327379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.327446 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.327586 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.327587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.327587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.327743 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.327844 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.327921 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.328003 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.21.28.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.124.keytab.
Entry for principal oryx/127.21.28.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.124.keytab.
I20260808 06:57:26.352227 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.124:39025
--webserver_interface=127.21.28.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.124.keytab
--principal=oryx/127.21.28.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:37545,127.21.28.125:40205,127.21.28.124:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:26.432722 31410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:26.432907 31410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:26.432927 31410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:26.433979 31410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:26.434007 31410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:26.434019 31410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:26.434031 31410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:26.434046 31410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:26.435487 31410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:37545,127.21.28.125:40205,127.21.28.124:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.124.keytab
--principal=oryx/127.21.28.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.124:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.28.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:26.435689 31410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:26.435863 31410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:26.440385 31419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.440507 31410 server_base.cc:1034] running on GCE node
W20260808 06:57:26.440402 31422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.440625 31424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.440834 31410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:26.441093 31410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:26.442225 31410 hybrid_clock.cc:648] HybridClock initialized: now 1786172246442214 us; error 23 us; skew 500 ppm
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:26.444473 31410 init.cc:377] Logged in from keytab as oryx/127.21.28.124@KRBTEST.COM (short username oryx)
I20260808 06:57:26.444845 31410 webserver.cc:460] Webserver started at http://127.21.28.124:41267/ using document root <none> and password file <none>
I20260808 06:57:26.445036 31410 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.445083 31410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.445205 31410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:26.446153 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data/instance:
uuid: "35b386fafcdc4068a44c02ef082a8246"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "fa6be2989b897a007b7acca50ae4fb03"
server_key_iv: "d833770a8f9c17673493623e249935e4"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.446466 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal/instance:
uuid: "35b386fafcdc4068a44c02ef082a8246"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "fa6be2989b897a007b7acca50ae4fb03"
server_key_iv: "d833770a8f9c17673493623e249935e4"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.447693 31410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:26.448347 31433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.448487 31410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:26.448570 31410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
uuid: "35b386fafcdc4068a44c02ef082a8246"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "fa6be2989b897a007b7acca50ae4fb03"
server_key_iv: "d833770a8f9c17673493623e249935e4"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.448630 31410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.468361 31410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:26.469918 31410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:26.470043 31410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:26.474279 31410 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.124:39025
I20260808 06:57:26.474323 31511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.124:39025 every 8 connection(s)
I20260808 06:57:26.474668 31410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
I20260808 06:57:26.475576 31512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:26.476573 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 31410
I20260808 06:57:26.477355 31512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.124@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.28.124@KRBTEST.COM
I20260808 06:57:26.485385 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.478387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.21.28.126:37545)
0808 06:57:26.478442 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.478446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.478475 (+ 29us) server_negotiation.cc:407] Connection header received
0808 06:57:26.478547 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.478548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.478559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.478603 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.479171 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.479650 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.480601 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.480783 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.480829 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.482839 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.482841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.482841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.482847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.484099 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.484517 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.484518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.484519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.484572 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.484778 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.484779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.484779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.484901 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.484976 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.485219 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.485294 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:26.485612 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.478233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:45996)
0808 06:57:26.478374 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.478422 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.478491 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.478629 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.478631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.478976 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.479082 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.479087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.479773 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.479774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.480530 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.480532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.480864 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.480865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.480927 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.481451 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.481459 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.482759 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.484169 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.484175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.484177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.484454 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.484624 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.484625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.484625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.484723 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.484980 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.484985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.485092 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.485193 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.485335 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260808 06:57:26.486057 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.477127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.21.28.124:39025)
0808 06:57:26.477300 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.477308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.477331 (+ 23us) server_negotiation.cc:407] Connection header received
0808 06:57:26.477872 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.477879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.477941 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.478066 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.478674 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.479761 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.480453 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.480806 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.480914 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.483344 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.483350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.483351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.483356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.484821 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.485147 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.485149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.485150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.485207 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.485386 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.485386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.485387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.485546 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.485682 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.485776 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.485847 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260808 06:57:26.486095 31512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.124@KRBTEST.COM for oryx/127.21.28.125@KRBTEST.COM
I20260808 06:57:26.491252 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.486334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.0.0.1:39684)
0808 06:57:26.486370 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.486378 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.486416 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.486546 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.486547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.486663 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.486696 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.486697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.487245 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.487245 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.487756 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.487757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.488049 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.488049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.488092 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.488408 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.488412 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.489408 (+ 996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.490591 (+ 1183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.490592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.490592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.490733 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.490868 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.490868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.490868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.490911 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.491067 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.491067 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.491097 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.491138 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.491197 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 06:57:26.491274 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.486401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.21.28.125:40205)
0808 06:57:26.486440 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.486442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.486449 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:26.486474 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.486475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.486486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.486536 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.486742 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.487194 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.487810 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.488001 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.488047 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.489487 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.489488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.489488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.489495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.490531 (+ 1036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.490783 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.490784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.490784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.490828 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.490944 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.490945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.490945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.491030 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.491070 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.491155 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.491214 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 06:57:26.491604 31512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.124@KRBTEST.COM for oryx/127.21.28.124@KRBTEST.COM
I20260808 06:57:26.496836 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.491828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:39025 (local address 127.0.0.1:41480)
0808 06:57:26.491871 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.491881 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.491936 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.492042 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.492042 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.492156 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.492177 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.492178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.492751 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.492752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.493203 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.493205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.493506 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.493507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.493551 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.493923 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.493927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.494940 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.496095 (+ 1155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.496095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.496096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.496279 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.496418 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.496418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.496418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.496475 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.496623 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.496623 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.496654 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.496695 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.496761 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 06:57:26.496894 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.491854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.21.28.124:39025)
0808 06:57:26.491880 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.491882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.491923 (+ 41us) server_negotiation.cc:407] Connection header received
0808 06:57:26.491984 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.491985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.491992 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.492035 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.492233 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.492690 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.493265 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.493454 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.493504 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.495015 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.495015 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.495016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.495021 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.496035 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.496332 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.496333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.496333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.496376 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.496512 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.496512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.496512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.496591 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.496630 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.496709 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.496803 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":13}
I20260808 06:57:26.498051 31512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Bootstrap starting.
I20260808 06:57:26.498831 31512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:26.499114 31512 log.cc:824] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:26.499809 31512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: No bootstrap required, opened a new log
I20260808 06:57:26.501191 31512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } }
I20260808 06:57:26.501307 31512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:26.501339 31512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b386fafcdc4068a44c02ef082a8246, State: Initialized, Role: FOLLOWER
I20260808 06:57:26.501467 31512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } }
I20260808 06:57:26.501756 31512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:26.501719 31523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } } }
I20260808 06:57:26.501808 31523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 06:57:26.503822 31534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:57:26.503867 31534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:57:26.505975 31410 master_runner.cc:428] Detected that this master 35b386fafcdc4068a44c02ef082a8246 is joining an existing cluster
I20260808 06:57:26.506016 31410 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.28.124:39025
I20260808 06:57:26.513350 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.506984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.21.28.126:37545)
0808 06:57:26.507026 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.507028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.507137 (+ 109us) server_negotiation.cc:407] Connection header received
0808 06:57:26.507241 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.507242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.507253 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.507311 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.507743 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.508502 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.509569 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.509845 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.509886 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.511029 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.511030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.511030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.511036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.512370 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.512743 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.512743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.512744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.512786 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.512958 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.512958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.512958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.513046 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.513088 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.513236 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.513288 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:26.513662 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.506935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46008)
0808 06:57:26.507080 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.507090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.507163 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.507356 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.507358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.507554 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.507623 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.507625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.508594 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.508595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.509469 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.509472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.509987 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.509988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.510049 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.510504 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.510508 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.510934 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.512444 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.512445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.512446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.512677 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.512824 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.512825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.512825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.512906 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.513094 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.513094 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.513136 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.513223 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.513318 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172245, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.126@KRBTEST.COM for oryx/127.21.28.124@KRBTEST.COM
I20260808 06:57:26.535133 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.515468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:39025 (local address 127.0.0.1:41488)
0808 06:57:26.515514 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.515525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.515564 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.515796 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.515797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.515967 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.516007 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.516009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.516589 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.516590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.517389 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.517391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.517696 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.517697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.517760 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.518055 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.518059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.519314 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.534021 (+ 14707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.534023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.534023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.534241 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.534499 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.534500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.534500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.534564 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.534830 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.534831 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.534875 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.534935 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.535032 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 06:57:26.535554 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.515518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.21.28.124:39025)
0808 06:57:26.515656 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.515658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.515664 (+ 6us) server_negotiation.cc:407] Connection header received
0808 06:57:26.515709 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.515709 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.515728 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.515783 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.516068 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.516526 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.517454 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.517638 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.517677 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.519401 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.519403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.519403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.519409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.533925 (+ 14516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.534369 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.534370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.534370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.534425 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.534633 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.534633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.534634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.534759 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.535295 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.535335 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.535411 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260808 06:57:26.535794 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.506980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.0.0.1:39692)
0808 06:57:26.507387 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.507395 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.507454 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.507605 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.507606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.507866 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.507941 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.507943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.508830 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.508831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.509680 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.509682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.510474 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.510475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.510531 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.532306 (+ 21775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.532311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.532908 (+ 597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.534593 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.534594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.534595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.534861 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.535105 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.535106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.535106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.535178 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.535445 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.535446 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.535500 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.535561 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.535662 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260808 06:57:26.536082 30447 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.28.124:39025: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "35b386fafcdc4068a44c02ef082a8246"
member_type: NON_VOTER
last_known_addr {
host: "127.21.28.124"
port: 39025
}
attrs {
promote: true
}
}
dest_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062"
cas_config_opid_index: 5
I20260808 06:57:26.536167 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.507300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.21.28.125:40205)
0808 06:57:26.507338 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.507340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.507459 (+ 119us) server_negotiation.cc:407] Connection header received
0808 06:57:26.507501 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.507501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.507513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.507561 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.508032 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.508748 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.509757 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.510062 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.510098 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.532998 (+ 22900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.533000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.533000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.533007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.534509 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.534975 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.534976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.534977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.535037 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.535240 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.535241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.535242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.535368 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.535421 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.535984 (+ 563us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.536076 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:26.536346 30447 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } }
I20260808 06:57:26.536760 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.507279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.21.28.124:39025)
0808 06:57:26.507322 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.507323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.507582 (+ 259us) server_negotiation.cc:407] Connection header received
0808 06:57:26.507650 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.507651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.507658 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.507719 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.508035 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.508728 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.509733 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.509989 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.510023 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.532749 (+ 22726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.532750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.532750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.532756 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.534804 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.535121 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.535122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.535122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.535219 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.536157 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.536158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.536158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.536262 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.536299 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.536621 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.536690 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 06:57:26.536813 31280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 35b386fafcdc4068a44c02ef082a8246 (127.21.28.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.537226 31543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.507135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:39025 (local address 127.0.0.1:41484)
0808 06:57:26.507539 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.507547 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.507599 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.507767 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.507767 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.507923 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.507969 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.507970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.508805 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.508805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.509652 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.509655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.510086 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.510087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.510136 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.532179 (+ 22043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.532184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.532659 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.534871 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.534872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.534872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.535052 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.536021 (+ 969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.536022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.536022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.536089 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.536485 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.536486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.536526 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.537039 (+ 513us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.537132 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":81,"threads_started":1}
I20260808 06:57:26.538014 31280 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.28.124:39025
I20260808 06:57:26.538089 31280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.538146 31280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:26.538370 31410 master.cc:537] Master@127.21.28.124:39025 shutting down...
I20260808 06:57:26.538357 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 30539
I20260808 06:57:26.541642 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.536917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.0.0.1:39694)
0808 06:57:26.536988 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.536997 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.537032 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.538111 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.538112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.538114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:26.538149 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.538150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.539123 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.539124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.540655 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.540659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.541450 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.541451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.541512 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.541537 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.541570 (+ 33us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.541597 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 06:57:26.541942 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.537867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.21.28.125:40205)
0808 06:57:26.537918 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.537920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.537927 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:26.537945 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.537945 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.537968 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.538023 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:26.538250 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.538991 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.540841 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.541309 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.541362 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.541422 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.541860 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.541892 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:57:26.554396 31349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 LEARNER]: Refusing update from remote peer e1fa7e28f9544dd1a57bfbf8bb65d062: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 06:57:26.555087 31279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260808 06:57:26.555801 30425 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 -> Peer 35b386fafcdc4068a44c02ef082a8246 (127.21.28.124:39025): Couldn't send request to peer 35b386fafcdc4068a44c02ef082a8246. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 06:57:26.556006 31349 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 35b386fafcdc4068a44c02ef082a8246 (127.21.28.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.557479 31390 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1fa7e28f9544dd1a57bfbf8bb65d062. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.557574 31390 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: This master's current role is: LEARNER
I20260808 06:57:26.557605 31390 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: NON_VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: true } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.557636 31390 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: This master's current role is: LEARNER
I20260808 06:57:26.558059 31279 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: attempting to promote NON_VOTER cffb51c0521e4dec9fb886b67223f949 to VOTER
I20260808 06:57:26.558254 31410 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:26.558259 31279 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } }
I20260808 06:57:26.558347 31410 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:26.558391 31410 tablet_replica.cc:330] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: stopping tablet replica
I20260808 06:57:26.558773 31349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 LEARNER]: Refusing update from remote peer e1fa7e28f9544dd1a57bfbf8bb65d062: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 06:57:26.558945 31280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 06:57:26.559286 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:38769
--local_ip_for_outbound_sockets=127.21.28.65
--tserver_master_addrs=127.21.28.126:37545
--webserver_port=35279
--webserver_interface=127.21.28.65
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab
--principal=oryx/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205,127.21.28.124:39025 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:26.559710 30425 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 -> Peer 35b386fafcdc4068a44c02ef082a8246 (127.21.28.124:39025): Couldn't send request to peer 35b386fafcdc4068a44c02ef082a8246. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 06:57:26.560098 31280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cffb51c0521e4dec9fb886b67223f949 (127.21.28.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.561080 31280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.561168 31280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:26.561385 31279 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.561491 31279 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:26.561510 31349 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cffb51c0521e4dec9fb886b67223f949 (127.21.28.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.562784 31552 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.562863 31552 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:57:26.574050 31410 master.cc:552] Master@127.21.28.124:39025 shutdown complete.
I20260808 06:57:26.574102 31410 master_runner.cc:305] Clearing existing system tablet
I20260808 06:57:26.574227 31410 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.574280 31410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.574975 31410 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:26.575487 31556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.575625 31410 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.575719 31410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.575747 31410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
uuid: "35b386fafcdc4068a44c02ef082a8246"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "fa6be2989b897a007b7acca50ae4fb03"
server_key_iv: "d833770a8f9c17673493623e249935e4"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.576491 31410 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:26.578001 31410 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 06:57:26.578078 31410 log.cc:1196] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 06:57:26.578274 31410 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Deleting consensus metadata
I20260808 06:57:26.578382 31410 master_runner.cc:315] Copying system tablet from 127.21.28.126:37545
I20260808 06:57:26.578996 31410 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: tablet copy: Beginning tablet copy session from remote peer at address 127.21.28.126:37545
I20260808 06:57:26.586580 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.579975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.21.28.126:37545)
0808 06:57:26.580018 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.580020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.580112 (+ 92us) server_negotiation.cc:407] Connection header received
0808 06:57:26.580163 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.580164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.580175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.580242 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.580603 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.581359 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.582372 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.582656 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.582721 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.583810 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.583811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.583811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.583819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.585387 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.585780 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.585781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.585781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.585845 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.586053 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.586054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.586054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.586182 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.586243 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.586408 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.586489 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:26.586588 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.579918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46022)
0808 06:57:26.580047 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.580057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.580115 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.580253 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.580254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.580468 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.580525 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.580528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.581443 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.581445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.582292 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.582295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.582732 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.582733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.582807 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.583286 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.583291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.583712 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.585475 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.585476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.585476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.585692 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.585914 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.585915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.585916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.585989 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.586249 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.586250 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.586302 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.586379 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.586490 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260808 06:57:26.587128 30472 tablet_copy_service.cc:140] P e1fa7e28f9544dd1a57bfbf8bb65d062: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 35b386fafcdc4068a44c02ef082a8246 ({username='oryx', principal='oryx/127.21.28.124@KRBTEST.COM'} at 127.0.0.1:46022)
I20260808 06:57:26.587185 30472 tablet_copy_service.cc:161] P e1fa7e28f9544dd1a57bfbf8bb65d062: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 35b386fafcdc4068a44c02ef082a8246 at {username='oryx', principal='oryx/127.21.28.124@KRBTEST.COM'} at 127.0.0.1:46022: session id = 35b386fafcdc4068a44c02ef082a8246-00000000000000000000000000000000
I20260808 06:57:26.587759 30472 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 06:57:26.588223 31410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 06:57:26.589614 31410 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: tablet copy: Starting download of 0 data blocks...
I20260808 06:57:26.589756 31410 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: tablet copy: Starting download of 1 WAL segments...
I20260808 06:57:26.590517 31410 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 06:57:26.591392 30472 tablet_copy_service.cc:342] P e1fa7e28f9544dd1a57bfbf8bb65d062: Request end of tablet copy session 35b386fafcdc4068a44c02ef082a8246-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.28.124@KRBTEST.COM'} at 127.0.0.1:46022
I20260808 06:57:26.591441 30472 tablet_copy_service.cc:433] P e1fa7e28f9544dd1a57bfbf8bb65d062: ending tablet copy session 35b386fafcdc4068a44c02ef082a8246-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 35b386fafcdc4068a44c02ef082a8246
I20260808 06:57:26.593858 31410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:26.594942 31566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.596640 31410 server_base.cc:1034] running on GCE node
W20260808 06:57:26.596838 31569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.604750 31567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.605067 31410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:26.605334 31410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:26.606485 31410 hybrid_clock.cc:648] HybridClock initialized: now 1786172246606465 us; error 46 us; skew 500 ppm
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:26.608022 31410 init.cc:377] Logged in from keytab as oryx/127.21.28.124@KRBTEST.COM (short username oryx)
I20260808 06:57:26.608290 31410 webserver.cc:460] Webserver started at http://127.21.28.124:42331/ using document root <none> and password file <none>
I20260808 06:57:26.608404 31410 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.608453 31410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.609185 31410 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:26.609797 31581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.609975 31410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.610028 31410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
uuid: "35b386fafcdc4068a44c02ef082a8246"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "fa6be2989b897a007b7acca50ae4fb03"
server_key_iv: "d833770a8f9c17673493623e249935e4"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.610087 31410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.629889 31410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:26.631771 31410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:26.635138 31410 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.124:39025
I20260808 06:57:26.635300 31660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.124:39025 every 8 connection(s)
I20260808 06:57:26.635474 31410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
I20260808 06:57:26.636772 31661 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:57:26.637326 31661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Bootstrap starting.
I20260808 06:57:26.640393 31661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:57:26.640595 31661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Bootstrap complete.
I20260808 06:57:26.640796 31661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } }
I20260808 06:57:26.640888 31661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 35b386fafcdc4068a44c02ef082a8246, State: Initialized, Role: LEARNER
I20260808 06:57:26.640935 31661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } }
I20260808 06:57:26.641095 31661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:26.641116 31666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:26.641316 31666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: This master's current role is: LEARNER
I20260808 06:57:26.643198 31682 catalog_manager.cc:1259] Loaded cluster ID: 28a6f53c4a714db3aa2fb5b40ce56110
I20260808 06:57:26.643281 31682 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: loading cluster ID for follower catalog manager: success
I20260808 06:57:26.644428 31682 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: acquiring CA information for follower catalog manager: success
I20260808 06:57:26.644862 31682 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 06:57:26.652384 31553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:26.652567 31553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:26.652588 31553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:26.653690 31553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:26.653728 31553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:26.653752 31553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:26.655176 31553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab
--principal=oryx/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:38769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=35279
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205,127.21.28.124:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:26.655385 31553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:26.655593 31553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:26.660712 31689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.660741 31688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.660975 31553 server_base.cc:1034] running on GCE node
W20260808 06:57:26.661067 31691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.661280 31553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:26.661561 31553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:26.662729 31553 hybrid_clock.cc:648] HybridClock initialized: now 1786172246662709 us; error 36 us; skew 500 ppm
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:26.666046 31553 init.cc:377] Logged in from keytab as oryx/127.21.28.65@KRBTEST.COM (short username oryx)
I20260808 06:57:26.666525 31553 webserver.cc:460] Webserver started at http://127.21.28.65:35279/ using document root <none> and password file <none>
I20260808 06:57:26.666785 31553 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.666854 31553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.668853 31553 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:26.669603 31700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.669770 31553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:26.669847 31553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "c6350088346744faacf9843062dd1e2b"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "d8b11085f38639b25854e6009cad0203"
server_key_iv: "b5de0819dd8e4109ff08799585315e65"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.670233 31553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.680488 31553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:26.682492 31553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:26.682617 31553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:26.682875 31553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:26.683166 31553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:26.683214 31553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.683243 31553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:26.683259 31553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.696403 31553 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:38769
I20260808 06:57:26.696498 31883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:38769 every 8 connection(s)
I20260808 06:57:26.696811 31553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.124@KRBTEST.COM
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.125@KRBTEST.COM
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:26.705685 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 31553
I20260808 06:57:26.705837 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 30725
I20260808 06:57:26.709038 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.697620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:47803 (local address 127.21.28.125:40205)
0808 06:57:26.697677 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.697680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.698134 (+ 454us) server_negotiation.cc:407] Connection header received
0808 06:57:26.699290 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.699292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.699304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.699351 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.700329 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.701088 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702438 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.702691 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702746 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.705519 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.705520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.705521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.705529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.707430 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.707950 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.707951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.707951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.708013 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.708291 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.708292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.708293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.708442 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.708494 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.708850 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.708942 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:26.709192 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.698066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:42511 (local address 127.21.28.126:37545)
0808 06:57:26.698120 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.698122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.698396 (+ 274us) server_negotiation.cc:407] Connection header received
0808 06:57:26.699338 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.699340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.699352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.699384 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.700375 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.700860 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702491 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.702779 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702845 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.706027 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.706028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.706029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.706036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.707568 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.708075 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.708076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.708077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.708147 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.708368 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.708369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.708369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.708519 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.708579 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.708961 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.709067 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 06:57:26.709940 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.697776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:46083 (local address 127.21.28.124:39025)
0808 06:57:26.698054 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.698057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.698066 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:26.699340 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.699341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.699355 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.699426 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.700357 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.701296 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702272 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.702528 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702590 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.705019 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.705021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.705021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.705028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.708731 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.709078 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.709079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.709079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.709142 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.709338 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.709339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.709339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.709491 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.709532 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.709732 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.709820 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":54,"threads_started":1}
I20260808 06:57:26.710095 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.697968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.65:42511)
0808 06:57:26.698325 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.698334 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.699200 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.699505 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.699506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.699948 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.700237 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.700244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.701015 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.701016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.702396 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.702398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.702845 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.702930 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.703757 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.703762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.705936 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.707693 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.707694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.707695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.707979 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.708223 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.708224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.708224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.708289 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.708616 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.708616 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.708710 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.709355 (+ 645us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.709682 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":24,"thread_start_us":230,"threads_started":1}
I20260808 06:57:26.710283 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.697555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:39025 (local address 127.21.28.65:46083)
0808 06:57:26.697905 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.697949 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.699186 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.699498 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.699499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.699936 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.700237 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.700244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.701367 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.701368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.702195 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.702197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.702600 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.702843 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.703322 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.703330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.704934 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.708812 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.708813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.708813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.708995 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.709214 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.709215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.709215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.709272 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.709583 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.709584 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.709627 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.710051 (+ 424us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.710143 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":6,"thread_start_us":180,"threads_started":1}
I20260808 06:57:26.710487 31891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.697572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.21.28.65:47803)
0808 06:57:26.697975 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.697984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.699191 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.699476 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.699480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.699892 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.700236 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.700241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.701162 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.701162 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.702355 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.702358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.702754 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.702755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.702865 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.703440 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.703444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.705401 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.707518 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.707526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.707528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.707859 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.708082 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.708083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.708083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.708212 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.708529 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.708536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.708703 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.709768 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.709865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":63,"thread_start_us":165,"threads_started":1}
I20260808 06:57:26.710816 31885 heartbeater.cc:347] Connected to a master server at 127.21.28.126:37545
I20260808 06:57:26.710897 31886 heartbeater.cc:347] Connected to a master server at 127.21.28.125:40205
I20260808 06:57:26.710963 31886 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:26.710963 31885 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:26.711171 31886 heartbeater.cc:511] Master 127.21.28.125:40205 requested a full tablet report, sending...
I20260808 06:57:26.711175 31885 heartbeater.cc:511] Master 127.21.28.126:37545 requested a full tablet report, sending...
I20260808 06:57:26.711594 30447 ts_manager.cc:194] Re-registered known tserver with Master: c6350088346744faacf9843062dd1e2b (127.21.28.65:38769)
I20260808 06:57:26.711730 31336 ts_manager.cc:194] Registered new tserver with Master: c6350088346744faacf9843062dd1e2b (127.21.28.65:38769)
I20260808 06:57:26.711809 31884 heartbeater.cc:347] Connected to a master server at 127.21.28.124:39025
I20260808 06:57:26.711858 31884 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:26.711934 31884 heartbeater.cc:511] Master 127.21.28.124:39025 requested a full tablet report, sending...
I20260808 06:57:26.712083 30447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:42511
I20260808 06:57:26.712339 31604 ts_manager.cc:194] Registered new tserver with Master: c6350088346744faacf9843062dd1e2b (127.21.28.65:38769)
I20260808 06:57:26.720927 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:46533
--local_ip_for_outbound_sockets=127.21.28.66
--tserver_master_addrs=127.21.28.126:37545
--webserver_port=46875
--webserver_interface=127.21.28.66
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab
--principal=oryx/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205,127.21.28.124:39025 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:26.800943 31894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:26.801126 31894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:26.801148 31894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:26.802209 31894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:26.802242 31894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:26.802268 31894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:26.803632 31894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab
--principal=oryx/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:46533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=46875
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205,127.21.28.124:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:26.803826 31894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:26.804006 31894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:26.808684 31906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.808689 31904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.808745 31894 server_base.cc:1034] running on GCE node
W20260808 06:57:26.808709 31902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.809058 31894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:26.809291 31894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:26.810420 31894 hybrid_clock.cc:648] HybridClock initialized: now 1786172246810400 us; error 36 us; skew 500 ppm
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:26.812783 31894 init.cc:377] Logged in from keytab as oryx/127.21.28.66@KRBTEST.COM (short username oryx)
I20260808 06:57:26.813159 31894 webserver.cc:460] Webserver started at http://127.21.28.66:46875/ using document root <none> and password file <none>
I20260808 06:57:26.813339 31894 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.813407 31894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.814565 31894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:26.815070 31916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.815208 31894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.815289 31894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "e96629b8965148f98b194c92e1997d6a"
format_stamp: "Formatted at 2026-08-08 06:57:25 on dist-test-slave-nj21"
server_key: "7de86a47fd8d91f74ab3a0b2f645131e"
server_key_iv: "48b43c8973ccd0bac4369d3d59a5b4ba"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.815582 31894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.832576 31894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:26.834197 31894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:26.834316 31894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:26.834519 31894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:26.834786 31894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:26.834817 31894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.834844 31894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:26.834862 31894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.841215 31894 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:46533
I20260808 06:57:26.841251 32087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:46533 every 8 connection(s)
I20260808 06:57:26.841615 31894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:26.845578 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 31894
I20260808 06:57:26.845739 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 30922
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.124@KRBTEST.COM
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.125@KRBTEST.COM
I20260808 06:57:26.868127 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:33083
--local_ip_for_outbound_sockets=127.21.28.67
--tserver_master_addrs=127.21.28.126:37545
--webserver_port=32859
--webserver_interface=127.21.28.67
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab
--principal=oryx/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205,127.21.28.124:39025 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:57:26.868970 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.842633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:45053 (local address 127.21.28.126:37545)
0808 06:57:26.842673 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.842674 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.842814 (+ 140us) server_negotiation.cc:407] Connection header received
0808 06:57:26.843917 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.843918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.843929 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.843982 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.844542 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.845023 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846447 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.846741 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846781 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.864967 (+ 18186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.864969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.864969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.864977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.867330 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.867923 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.867925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.867925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.867988 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.868274 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.868276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.868276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.868417 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.868457 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.868803 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.868881 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 06:57:26.869011 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.842313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:39609 (local address 127.21.28.124:39025)
0808 06:57:26.842368 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.842370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.842736 (+ 366us) server_negotiation.cc:407] Connection header received
0808 06:57:26.843784 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.843786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.843799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.843846 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.844517 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.845294 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846437 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.846754 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846789 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.865824 (+ 19035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.865826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.865826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.865834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.867287 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.867938 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.867939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.867940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.868001 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.868416 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.868417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.868417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.868544 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.868585 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.868831 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.868924 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 06:57:26.869009 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.842269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:33773 (local address 127.21.28.125:40205)
0808 06:57:26.842384 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.842386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.842603 (+ 217us) server_negotiation.cc:407] Connection header received
0808 06:57:26.843721 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.843723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.843735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.843782 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.844549 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.845291 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846440 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.846817 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846865 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.866136 (+ 19271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.866138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.866139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.866147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.867570 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.867990 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.867991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.867992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.868046 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.868393 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.868394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.868394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.868543 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.868585 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.868831 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.868914 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 06:57:26.869509 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.842552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.66:45053)
0808 06:57:26.842777 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.842783 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.843598 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.843995 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.843996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.844265 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.844433 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.844439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.845121 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.845122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.846345 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.846347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.846868 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.847002 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.848542 (+ 1540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.848553 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.864868 (+ 16315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.867430 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.867431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.867432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.867836 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.868058 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.868059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.868059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.868193 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.868502 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.868508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.868665 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.869048 (+ 383us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.869216 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":45,"thread_start_us":61,"threads_started":1}
I20260808 06:57:26.869589 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.842251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.21.28.66:33773)
0808 06:57:26.842485 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.842525 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.843592 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.843857 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.843861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.844256 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.844433 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.844439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.845398 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.845399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.846328 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.846331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.846966 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.847047 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.850214 (+ 3167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.850219 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.866016 (+ 15797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.867654 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.867655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.867655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.867912 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.868112 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.868112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.868113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.868321 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.868634 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.868635 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.868683 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.869300 (+ 617us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.869419 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260808 06:57:26.869890 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.842257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:39025 (local address 127.21.28.66:39609)
0808 06:57:26.842684 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.842693 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.843599 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.843863 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.843864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.844240 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.844432 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.844436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.845370 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.845370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.846339 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.846343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.846866 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.846867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.847000 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.849590 (+ 2590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.849595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.865739 (+ 16144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.867376 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.867384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.867386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.867840 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.868071 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.868072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.868072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.868193 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.868634 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.868635 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.868683 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.869336 (+ 653us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.869487 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":61,"thread_start_us":87,"threads_started":1}
I20260808 06:57:26.870003 32093 heartbeater.cc:347] Connected to a master server at 127.21.28.125:40205
I20260808 06:57:26.870083 32092 heartbeater.cc:347] Connected to a master server at 127.21.28.126:37545
I20260808 06:57:26.870097 32093 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:26.870139 32092 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:26.870311 32092 heartbeater.cc:511] Master 127.21.28.126:37545 requested a full tablet report, sending...
I20260808 06:57:26.870345 32091 heartbeater.cc:347] Connected to a master server at 127.21.28.124:39025
I20260808 06:57:26.870381 32091 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:26.870445 32091 heartbeater.cc:511] Master 127.21.28.124:39025 requested a full tablet report, sending...
I20260808 06:57:26.870309 32093 heartbeater.cc:511] Master 127.21.28.125:40205 requested a full tablet report, sending...
I20260808 06:57:26.870698 30447 ts_manager.cc:194] Re-registered known tserver with Master: e96629b8965148f98b194c92e1997d6a (127.21.28.66:46533)
I20260808 06:57:26.870767 31336 ts_manager.cc:194] Registered new tserver with Master: e96629b8965148f98b194c92e1997d6a (127.21.28.66:46533)
I20260808 06:57:26.870783 31604 ts_manager.cc:194] Registered new tserver with Master: e96629b8965148f98b194c92e1997d6a (127.21.28.66:46533)
I20260808 06:57:26.871174 30447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:45053
W20260808 06:57:26.951406 32103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:26.951594 32103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:26.951615 32103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:26.952823 32103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:26.952867 32103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:26.952895 32103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:26.954325 32103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab
--principal=oryx/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:33083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=32859
--tserver_master_addrs=127.21.28.126:37545,127.21.28.125:40205,127.21.28.124:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:26.954641 32103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:26.954905 32103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:26.959563 32115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.959616 32109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:26.959573 32112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:26.959769 32103 server_base.cc:1034] running on GCE node
I20260808 06:57:26.959929 32103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:26.960115 32103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:26.961249 32103 hybrid_clock.cc:648] HybridClock initialized: now 1786172246961238 us; error 27 us; skew 500 ppm
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:26.963521 32103 init.cc:377] Logged in from keytab as oryx/127.21.28.67@KRBTEST.COM (short username oryx)
I20260808 06:57:26.963943 32103 webserver.cc:460] Webserver started at http://127.21.28.67:32859/ using document root <none> and password file <none>
I20260808 06:57:26.964109 32103 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:26.964157 32103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:26.965466 32103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:26.965953 32124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.966091 32103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.966171 32103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "4792a68b34154ac9872393a4934b71d1"
format_stamp: "Formatted at 2026-08-08 06:57:26 on dist-test-slave-nj21"
server_key: "96d8e6f180a20467a9537f7378974d78"
server_key_iv: "3b07708a334e3eba57bc851866daffe1"
server_key_version: "encryptionkey@0"
I20260808 06:57:26.966465 32103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:26.978585 32103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:26.980136 32103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:26.980257 32103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:26.980454 32103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:26.980713 32103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:26.980746 32103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.980772 32103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:26.980795 32103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:26.986560 32103 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:33083
I20260808 06:57:26.986618 32297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:33083 every 8 connection(s)
I20260808 06:57:26.986899 32103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.124@KRBTEST.COM
I20260808 06:57:26.994141 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 32103
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.125@KRBTEST.COM
Aug 08 06:57:26 dist-test-slave-nj21 krb5kdc[30385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172246, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:26.998744 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.987830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:39025 (local address 127.21.28.67:33619)
0808 06:57:26.988011 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.988020 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.988857 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.989081 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.989082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.989464 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.989692 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.989698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.990406 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.990407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.991574 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.991576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.992098 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.992099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.992203 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.992643 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.992650 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.994353 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.996557 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.996564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.996566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.996847 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.997711 (+ 864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.997712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.997712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.997830 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.998129 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.998136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.998274 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.998367 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.998521 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260808 06:57:26.998842 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.987534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:34283 (local address 127.21.28.126:37545)
0808 06:57:26.987578 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.987579 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.987833 (+ 254us) server_negotiation.cc:407] Connection header received
0808 06:57:26.988970 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.988972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.988985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.989016 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.989809 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.990547 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.992439 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.992738 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.992802 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.996004 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.996005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.996006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.996013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.997728 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.998084 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.998085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.998085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.998147 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.998329 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.998330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.998330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.998448 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.998497 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.998671 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.998749 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:26.999183 32306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.987539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.67:34283)
0808 06:57:26.987719 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.987758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.988842 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.989081 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.989082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.989475 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.989743 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.989745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.991686 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.991687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.992332 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.992334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.992802 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.992803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.992865 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.993207 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.993212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.995892 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.997807 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.997808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.997808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.998009 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.998207 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.998208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.998208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.998263 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.998530 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.998530 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.998568 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 06:57:26.998945 (+ 377us) client_negotiation.cc:240] Negotiation successful
0808 06:57:26.999034 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260808 06:57:26.999426 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.988001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:33619 (local address 127.21.28.124:39025)
0808 06:57:26.988058 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.988060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.988069 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:26.988974 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.988976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.988987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.989016 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.989805 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.990309 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.991656 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.991991 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.992046 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.994448 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.994449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.994449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.994458 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.996495 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.997583 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.997584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.997585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.997646 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.997910 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.997911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.997911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.998043 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.998104 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:26.999256 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0808 06:57:26.999341 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:57:26.999567 32302 heartbeater.cc:347] Connected to a master server at 127.21.28.126:37545
I20260808 06:57:26.999696 32302 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:26.999883 32302 heartbeater.cc:511] Master 127.21.28.126:37545 requested a full tablet report, sending...
I20260808 06:57:27.000216 32300 heartbeater.cc:347] Connected to a master server at 127.21.28.124:39025
I20260808 06:57:27.000262 32300 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:27.000317 32300 heartbeater.cc:511] Master 127.21.28.124:39025 requested a full tablet report, sending...
I20260808 06:57:27.000334 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.987540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.21.28.67:46529)
0808 06:57:26.987813 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:26.987823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:26.988846 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:26.989065 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:26.989069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:26.989426 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:26.989692 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.989696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.990617 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.990618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.991574 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:26.991576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.992045 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.992045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:26.992206 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.992779 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:26.992783 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:26.994804 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:26.997310 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.997311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.997311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.997498 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.999111 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:26.999112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:26.999112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:26.999167 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:26.999912 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:26.999912 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:26.999951 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 06:57:27.000010 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 06:57:27.000100 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":7,"thread_start_us":163,"threads_started":1}
I20260808 06:57:27.000499 30447 ts_manager.cc:194] Re-registered known tserver with Master: 4792a68b34154ac9872393a4934b71d1 (127.21.28.67:33083)
I20260808 06:57:27.000675 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.987544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:46529 (local address 127.21.28.125:40205)
0808 06:57:26.987591 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.987593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.987873 (+ 280us) server_negotiation.cc:407] Connection header received
0808 06:57:26.988936 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.988938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.988950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.988992 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.989776 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.990537 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.991651 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.991965 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.992032 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:26.994922 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:26.994924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:26.994924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:26.994932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:26.997222 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.998985 (+ 1763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.998987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.998987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.999046 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:26.999686 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:26.999687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:26.999687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:26.999819 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:26.999876 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.000503 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.000588 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:27.000965 30447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:34283
I20260808 06:57:27.001384 31604 ts_manager.cc:194] Registered new tserver with Master: 4792a68b34154ac9872393a4934b71d1 (127.21.28.67:33083)
I20260808 06:57:27.001899 32303 heartbeater.cc:347] Connected to a master server at 127.21.28.125:40205
I20260808 06:57:27.001959 32303 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:27.002012 32303 heartbeater.cc:511] Master 127.21.28.125:40205 requested a full tablet report, sending...
I20260808 06:57:27.002285 31336 ts_manager.cc:194] Registered new tserver with Master: 4792a68b34154ac9872393a4934b71d1 (127.21.28.67:33083)
I20260808 06:57:27.004855 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.996019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.21.28.124:39025)
0808 06:57:26.996155 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.996157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.996542 (+ 385us) server_negotiation.cc:407] Connection header received
0808 06:57:26.996632 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.996633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.996644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.996680 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.997463 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.998170 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.999129 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.999403 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:26.999452 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.002126 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:27.002127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:27.002127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:27.002134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:27.003660 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.004090 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.004091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.004091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.004145 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.004309 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.004310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.004310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.004445 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:27.004547 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.004709 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.004770 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260808 06:57:27.005795 32319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.995643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.21.28.126:37545)
0808 06:57:26.995806 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:26.995808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:26.996350 (+ 542us) server_negotiation.cc:407] Connection header received
0808 06:57:26.996446 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:26.996448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:26.996460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:26.996543 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:26.997638 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:26.998333 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.000126 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.000415 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.000458 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.002274 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:27.002275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:27.002276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:27.002282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:27.004003 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.004372 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.004373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.004373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.004428 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.004889 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.004890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.004890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.005023 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:27.005116 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.005621 (+ 505us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.005698 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260808 06:57:27.007059 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:26.995584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.21.28.125:40205)
0808 06:57:27.000749 (+ 5165us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.000751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.000758 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:27.000776 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.000777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.000787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.000836 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:27.001345 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.002004 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.002919 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.003178 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.003244 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.004631 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:27.004633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:27.004633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:27.004639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:27.006078 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.006389 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.006391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.006391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.006479 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.006641 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.006642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.006642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.006738 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:27.006813 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.006916 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.006981 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5151,"thread_start_us":53,"threads_started":1}
I20260808 06:57:27.007308 30447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:27.009223 30447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:27.016463 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.012885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:33083 (local address 127.0.0.1:58794)
0808 06:57:27.013048 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:27.013055 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:27.013081 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:27.013626 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:27.013627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:27.013629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:27.013666 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.013668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.014617 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.014618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.015417 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.015420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.016265 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.016265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.016320 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.016324 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:27.016361 (+ 37us) client_negotiation.cc:240] Negotiation successful
0808 06:57:27.016395 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":130,"threads_started":1}
I20260808 06:57:27.016551 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.013212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.21.28.67:33083)
0808 06:57:27.013438 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.013442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.013451 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:27.013508 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.013510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.013525 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.013624 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:27.013742 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.014549 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.015476 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.016189 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.016261 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.016386 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.016412 (+ 26us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.016446 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
I20260808 06:57:27.016996 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.012817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46533 (local address 127.0.0.1:45774)
0808 06:57:27.012857 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:27.012866 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:27.012899 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:27.013465 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:27.013466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:27.013468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:27.013502 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.013504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.014733 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.014734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.015829 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.015832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.016781 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.016782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.016841 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.016844 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:27.016889 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 06:57:27.016938 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 06:57:27.017079 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.012994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.21.28.66:46533)
0808 06:57:27.013194 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.013199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.013208 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:27.013322 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.013324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.013344 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.013465 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:27.013578 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.014643 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.015909 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.016706 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.016770 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.016908 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.016934 (+ 26us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.016970 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":125,"threads_started":1}
I20260808 06:57:27.017472 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.013133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:38769 (local address 127.0.0.1:58478)
0808 06:57:27.013742 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:27.013752 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:27.013807 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:27.013968 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:27.013969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:27.013971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:27.014008 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.014009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.015283 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.015284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.016231 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.016233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.017238 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.017239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.017298 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.017301 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:27.017348 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 06:57:27.017386 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":101,"threads_started":1}
I20260808 06:57:27.017647 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.013246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.21.28.65:38769)
0808 06:57:27.013575 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.013580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.013821 (+ 241us) server_negotiation.cc:407] Connection header received
0808 06:57:27.013867 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.013870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.013885 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.013962 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:27.014063 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.015197 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.016313 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.017044 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.017108 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.017204 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.017374 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.017412 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260808 06:57:27.017788 32198 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff6dc89d946439d8ac44bbcc6a47c34 (DEFAULT_TABLE table=test-table [id=fbc03bdd22674c98ac74a98367a809d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:27.017951 31983 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff6dc89d946439d8ac44bbcc6a47c34 (DEFAULT_TABLE table=test-table [id=fbc03bdd22674c98ac74a98367a809d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:27.018213 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff6dc89d946439d8ac44bbcc6a47c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:27.018276 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff6dc89d946439d8ac44bbcc6a47c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:27.018281 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff6dc89d946439d8ac44bbcc6a47c34 (DEFAULT_TABLE table=test-table [id=fbc03bdd22674c98ac74a98367a809d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:27.018555 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff6dc89d946439d8ac44bbcc6a47c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:27.022162 32338 tablet_bootstrap.cc:492] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Bootstrap starting.
I20260808 06:57:27.022900 32340 tablet_bootstrap.cc:492] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Bootstrap starting.
I20260808 06:57:27.023451 32338 tablet_bootstrap.cc:654] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:27.023835 32339 tablet_bootstrap.cc:492] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: Bootstrap starting.
I20260808 06:57:27.024240 32340 tablet_bootstrap.cc:654] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:27.024264 32338 log.cc:824] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:27.024471 32339 tablet_bootstrap.cc:654] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:27.024720 32340 log.cc:824] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:27.025126 32338 tablet_bootstrap.cc:492] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: No bootstrap required, opened a new log
I20260808 06:57:27.025125 32339 log.cc:824] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:27.025228 32338 ts_tablet_manager.cc:1397] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260808 06:57:27.025499 32340 tablet_bootstrap.cc:492] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: No bootstrap required, opened a new log
I20260808 06:57:27.025565 32340 ts_tablet_manager.cc:1397] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 06:57:27.025830 32339 tablet_bootstrap.cc:492] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: No bootstrap required, opened a new log
I20260808 06:57:27.025884 32339 ts_tablet_manager.cc:1397] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:27.028108 32338 raft_consensus.cc:348] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.028268 32338 raft_consensus.cc:374] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:27.028309 32338 raft_consensus.cc:729] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6350088346744faacf9843062dd1e2b, State: Initialized, Role: FOLLOWER
I20260808 06:57:27.028435 32338 consensus_queue.cc:260] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.028710 32338 ts_tablet_manager.cc:1428] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 06:57:27.028748 31885 heartbeater.cc:503] Master 127.21.28.126:37545 was elected leader, sending a full tablet report...
I20260808 06:57:27.029017 32339 raft_consensus.cc:348] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.029156 32339 raft_consensus.cc:374] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:27.029189 32339 raft_consensus.cc:729] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4792a68b34154ac9872393a4934b71d1, State: Initialized, Role: FOLLOWER
I20260808 06:57:27.029358 32339 consensus_queue.cc:260] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.029660 32339 ts_tablet_manager.cc:1428] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260808 06:57:27.029939 32340 raft_consensus.cc:348] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.030049 32302 heartbeater.cc:503] Master 127.21.28.126:37545 was elected leader, sending a full tablet report...
I20260808 06:57:27.030072 32340 raft_consensus.cc:374] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:27.030103 32340 raft_consensus.cc:729] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e96629b8965148f98b194c92e1997d6a, State: Initialized, Role: FOLLOWER
I20260808 06:57:27.030206 32340 consensus_queue.cc:260] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.030644 32340 ts_tablet_manager.cc:1428] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260808 06:57:27.030826 32092 heartbeater.cc:503] Master 127.21.28.126:37545 was elected leader, sending a full tablet report...
I20260808 06:57:27.082729 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.080438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.21.28.124:39025)
0808 06:57:27.080497 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.080499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.080507 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:27.080533 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.080534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.080556 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.080612 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:27.080709 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.081194 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.082016 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.082417 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.082471 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.082530 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.082642 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.082672 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:57:27.082748 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.080333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:39025 (local address 127.0.0.1:41506)
0808 06:57:27.080377 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:27.080390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:27.080454 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:27.080619 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:27.080619 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:27.080621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:27.080666 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.080667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.081240 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.081240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.081959 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.081961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.082510 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.082511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.082569 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.082573 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:27.082637 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 06:57:27.082680 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260808 06:57:27.083165 31617 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260808 06:57:27.084144 31666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1fa7e28f9544dd1a57bfbf8bb65d062. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:27.084254 31666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: This master's current role is: LEARNER
W20260808 06:57:27.092295 32096 tablet.cc:2369] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:27.182119 32345 raft_consensus.cc:482] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:27.182243 32345 raft_consensus.cc:504] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.182682 32345 leader_election.cc:288] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e96629b8965148f98b194c92e1997d6a (127.21.28.66:46533), c6350088346744faacf9843062dd1e2b (127.21.28.65:38769)
I20260808 06:57:27.185535 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.182969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:44019 (local address 127.21.28.66:46533)
0808 06:57:27.183022 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.183024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.183033 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:27.183061 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.183062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.183071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.183122 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:27.183368 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.183878 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.184800 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.185154 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.185209 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.185219 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.185428 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.185464 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:57:27.185542 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.182941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:56929 (local address 127.21.28.65:38769)
0808 06:57:27.182984 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.182986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.183046 (+ 60us) server_negotiation.cc:407] Connection header received
0808 06:57:27.183106 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.183107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.183116 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.183167 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:27.183365 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.183867 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.184808 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.185167 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.185228 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.185237 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.185435 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.185476 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:27.185556 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.182847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:38769 (local address 127.21.28.67:56929)
0808 06:57:27.182987 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:27.182997 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:27.183050 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:27.183184 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:27.183184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:27.183186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:27.183300 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.183301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.183941 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.183942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.184723 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.184725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.185236 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.185237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.185362 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.185365 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:27.185413 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 06:57:27.185457 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":135}
I20260808 06:57:27.185621 32306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.182847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46533 (local address 127.21.28.67:44019)
0808 06:57:27.182902 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:27.182916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:27.182977 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:27.183132 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:27.183133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:27.183134 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:27.183293 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.183295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.183955 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.183957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.184718 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:27.184720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.185227 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.185228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:27.185355 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.185359 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:27.185404 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 06:57:27.185453 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":24}
I20260808 06:57:27.185930 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff6dc89d946439d8ac44bbcc6a47c34" candidate_uuid: "4792a68b34154ac9872393a4934b71d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6350088346744faacf9843062dd1e2b" is_pre_election: true
I20260808 06:57:27.186023 32017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff6dc89d946439d8ac44bbcc6a47c34" candidate_uuid: "4792a68b34154ac9872393a4934b71d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e96629b8965148f98b194c92e1997d6a" is_pre_election: true
I20260808 06:57:27.186110 31804 raft_consensus.cc:2454] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4792a68b34154ac9872393a4934b71d1 in term 0.
I20260808 06:57:27.186148 32017 raft_consensus.cc:2454] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4792a68b34154ac9872393a4934b71d1 in term 0.
I20260808 06:57:27.186378 32127 leader_election.cc:302] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4792a68b34154ac9872393a4934b71d1, c6350088346744faacf9843062dd1e2b, e96629b8965148f98b194c92e1997d6a; no voters:
I20260808 06:57:27.186565 32345 raft_consensus.cc:2781] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:27.186619 32345 raft_consensus.cc:482] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:27.186641 32345 raft_consensus.cc:3037] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:27.187462 32345 raft_consensus.cc:504] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.187582 32345 leader_election.cc:288] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [CANDIDATE]: Term 1 election: Requested vote from peers e96629b8965148f98b194c92e1997d6a (127.21.28.66:46533), c6350088346744faacf9843062dd1e2b (127.21.28.65:38769)
I20260808 06:57:27.187898 32017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff6dc89d946439d8ac44bbcc6a47c34" candidate_uuid: "4792a68b34154ac9872393a4934b71d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e96629b8965148f98b194c92e1997d6a"
I20260808 06:57:27.187985 32017 raft_consensus.cc:3037] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:27.188160 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff6dc89d946439d8ac44bbcc6a47c34" candidate_uuid: "4792a68b34154ac9872393a4934b71d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6350088346744faacf9843062dd1e2b"
I20260808 06:57:27.188238 31804 raft_consensus.cc:3037] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:27.189430 32017 raft_consensus.cc:2454] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4792a68b34154ac9872393a4934b71d1 in term 1.
I20260808 06:57:27.189601 31804 raft_consensus.cc:2454] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4792a68b34154ac9872393a4934b71d1 in term 1.
I20260808 06:57:27.189594 32127 leader_election.cc:302] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4792a68b34154ac9872393a4934b71d1, e96629b8965148f98b194c92e1997d6a; no voters:
I20260808 06:57:27.189778 32345 raft_consensus.cc:2781] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:27.189917 32345 raft_consensus.cc:686] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 1 LEADER]: Becoming Leader. State: Replica: 4792a68b34154ac9872393a4934b71d1, State: Running, Role: LEADER
I20260808 06:57:27.190006 32345 consensus_queue.cc:237] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:27.190820 30446 catalog_manager.cc:5495] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4792a68b34154ac9872393a4934b71d1 (127.21.28.67). New cstate: current_term: 1 leader_uuid: "4792a68b34154ac9872393a4934b71d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } health_report { overall_health: UNKNOWN } } }
I20260808 06:57:27.191541 31547 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: NON_VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: true } } }
I20260808 06:57:27.191634 31547 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:27.192320 32357 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: attempting to promote NON_VOTER 35b386fafcdc4068a44c02ef082a8246 to VOTER
I20260808 06:57:27.192451 32357 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } }
I20260808 06:57:27.192824 31349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Refusing update from remote peer e1fa7e28f9544dd1a57bfbf8bb65d062: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260808 06:57:27.193012 31547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 06:57:27.193138 31617 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 LEARNER]: Refusing update from remote peer e1fa7e28f9544dd1a57bfbf8bb65d062: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260808 06:57:27.193419 31547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 06:57:27.193656 31547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 35b386fafcdc4068a44c02ef082a8246 (127.21.28.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:27.193785 31349 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 35b386fafcdc4068a44c02ef082a8246 (127.21.28.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:27.194787 31547 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:27.194856 31547 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:27.194893 31547 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:27.194926 31547 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:27.196053 32360 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:27.196143 32360 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:57:27.196498 31280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:27.196583 31280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:27.196669 31617 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 35b386fafcdc4068a44c02ef082a8246 (127.21.28.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:27.197701 31666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:27.197785 31666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 06:57:27.197814 31889 tablet.cc:2369] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:27.198523 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.195894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.21.28.67:33083)
0808 06:57:27.195945 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.195947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.195955 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:27.195984 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.195984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.196001 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.196067 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:27.196239 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.196855 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.197843 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.198111 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.198171 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.198300 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:27.198386 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:27.198445 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.198458 (+ 13us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.198474 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:57:27.200296 32017 raft_consensus.cc:1261] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 1 FOLLOWER]: Refusing update from remote peer 4792a68b34154ac9872393a4934b71d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:27.200326 31804 raft_consensus.cc:1261] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 1 FOLLOWER]: Refusing update from remote peer 4792a68b34154ac9872393a4934b71d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:27.200562 32345 consensus_queue.cc:1035] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:27.200685 32353 consensus_queue.cc:1035] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:27.203357 32363 mvcc.cc:205] Tried to move back new op lower bound from 7316161524530229248 to 7316161524490846208. Current Snapshot: MvccSnapshot[applied={T|T < 7316161524530229248}]
I20260808 06:57:27.205540 32365 mvcc.cc:205] Tried to move back new op lower bound from 7316161524530229248 to 7316161524490846208. Current Snapshot: MvccSnapshot[applied={T|T < 7316161524530229248}]
I20260808 06:57:27.206025 30446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:27.206103 30446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:27.207347 30446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1fa7e28f9544dd1a57bfbf8bb65d062: Sending DeleteTablet for 3 replicas of tablet 6ff6dc89d946439d8ac44bbcc6a47c34
I20260808 06:57:27.207707 32198 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ff6dc89d946439d8ac44bbcc6a47c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:27 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:58794
I20260808 06:57:27.208007 32377 tablet_replica.cc:330] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: stopping tablet replica
I20260808 06:57:27.207990 31776 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ff6dc89d946439d8ac44bbcc6a47c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:27 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:58478
I20260808 06:57:27.208107 32377 raft_consensus.cc:2229] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:27.208194 32377 raft_consensus.cc:2258] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:27.208422 31983 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ff6dc89d946439d8ac44bbcc6a47c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:27 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:45774
I20260808 06:57:27.208668 32379 tablet_replica.cc:330] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: stopping tablet replica
I20260808 06:57:27.208771 32379 raft_consensus.cc:2229] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:27.208799 32381 tablet_replica.cc:330] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: stopping tablet replica
I20260808 06:57:27.208875 32379 raft_consensus.cc:2258] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:27.209056 32381 raft_consensus.cc:2229] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:27.209199 32381 raft_consensus.cc:2258] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:27.209270 32379 ts_tablet_manager.cc:1905] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:27.209579 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 30401
I20260808 06:57:27.209787 32381 ts_tablet_manager.cc:1905] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:27.215562 32379 ts_tablet_manager.cc:1918] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:27.215649 32379 log.cc:1196] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/6ff6dc89d946439d8ac44bbcc6a47c34
I20260808 06:57:27.215946 32379 ts_tablet_manager.cc:1939] T 6ff6dc89d946439d8ac44bbcc6a47c34 P c6350088346744faacf9843062dd1e2b: Deleting consensus metadata
I20260808 06:57:27.217432 32381 ts_tablet_manager.cc:1918] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:27.217514 32381 log.cc:1196] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/6ff6dc89d946439d8ac44bbcc6a47c34
I20260808 06:57:27.217763 32381 ts_tablet_manager.cc:1939] T 6ff6dc89d946439d8ac44bbcc6a47c34 P e96629b8965148f98b194c92e1997d6a: Deleting consensus metadata
I20260808 06:57:27.219957 32377 ts_tablet_manager.cc:1905] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:27.221522 32377 ts_tablet_manager.cc:1918] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:27.221589 32377 log.cc:1196] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/6ff6dc89d946439d8ac44bbcc6a47c34
I20260808 06:57:27.221805 32377 ts_tablet_manager.cc:1939] T 6ff6dc89d946439d8ac44bbcc6a47c34 P 4792a68b34154ac9872393a4934b71d1: Deleting consensus metadata
W20260808 06:57:27.224997 31702 connection.cc:535] server connection from 127.0.0.1:58478 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 06:57:27.225112 32125 connection.cc:535] server connection from 127.0.0.1:58794 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 06:57:27.225368 31922 connection.cc:535] server connection from 127.0.0.1:45774 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 06:57:27.229676 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.209281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.21.28.124:39025)
0808 06:57:27.209325 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.209326 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.222229 (+ 12903us) server_negotiation.cc:407] Connection header received
0808 06:57:27.222274 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.222275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.222288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.222336 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:27.222909 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.223408 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.224162 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.224365 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.224419 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.226263 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:27.226265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:27.226265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:27.226273 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:27.228247 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.228621 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.228623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.228623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.228684 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.228886 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.228887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.228887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.229004 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:27.229091 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.229502 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.229580 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 06:57:27.234580 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.227225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.21.28.124:39025)
0808 06:57:27.227264 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.227266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.227605 (+ 339us) server_negotiation.cc:407] Connection header received
0808 06:57:27.227736 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.227737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.227748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.227777 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:27.228726 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.229394 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.230305 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.230493 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.230544 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.231611 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:27.231612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:27.231613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:27.231619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:27.233442 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.233876 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.233877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.233878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.233935 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.234105 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.234106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.234106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.234230 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:27.234319 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.234416 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.234490 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:27.234750 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:27.227413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.21.28.125:40205)
0808 06:57:27.227464 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:27.227466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:27.227476 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:27.227527 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:27.227528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:27.227541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:27.227570 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:27.228258 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.229000 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.230049 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:27.230304 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:27.230343 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:27.231899 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:27.231901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:27.231901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:27.231908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:27.233399 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.233939 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.233941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.233941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.233996 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:27.234156 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:27.234157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:27.234157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:27.234294 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:27.234393 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:27.234578 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 06:57:27.234658 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 06:57:28.205191 32396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.204942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.66:51241)
0808 06:57:28.205079 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.205146 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260808 06:57:28.205191 32397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.204943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.65:46577)
0808 06:57:28.205065 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.205145 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
W20260808 06:57:28.205502 31885 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:37545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
W20260808 06:57:28.205571 32092 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:37545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
I20260808 06:57:28.206077 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.205808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.67:60869)
0808 06:57:28.205960 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.206035 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":52,"threads_started":1}
W20260808 06:57:28.206270 32302 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:37545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
I20260808 06:57:28.708046 32401 raft_consensus.cc:482] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1fa7e28f9544dd1a57bfbf8bb65d062)
I20260808 06:57:28.708212 32401 raft_consensus.cc:504] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } }
I20260808 06:57:28.708396 32402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1fa7e28f9544dd1a57bfbf8bb65d062)
I20260808 06:57:28.708506 32401 leader_election.cc:288] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1fa7e28f9544dd1a57bfbf8bb65d062 (127.21.28.126:37545), 35b386fafcdc4068a44c02ef082a8246 (127.21.28.124:39025)
I20260808 06:57:28.708535 32402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } }
I20260808 06:57:28.708793 32402 leader_election.cc:288] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1fa7e28f9544dd1a57bfbf8bb65d062 (127.21.28.126:37545), cffb51c0521e4dec9fb886b67223f949 (127.21.28.125:40205)
I20260808 06:57:28.708986 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.708641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46498)
0808 06:57:28.708880 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.708943 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":42,"threads_started":1}
W20260808 06:57:28.709103 31315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
I20260808 06:57:28.712137 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.709440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46512)
0808 06:57:28.712073 (+ 2633us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.712103 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2618}
W20260808 06:57:28.712241 31315 leader_election.cc:334] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1fa7e28f9544dd1a57bfbf8bb65d062 (127.21.28.126:37545): Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
I20260808 06:57:28.712378 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.709029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46510)
0808 06:57:28.709326 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.712340 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":40,"threads_started":1}
W20260808 06:57:28.712492 31583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
I20260808 06:57:28.712930 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.712835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46532)
0808 06:57:28.712874 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.712896 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 06:57:28.713001 31583 leader_election.cc:334] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1fa7e28f9544dd1a57bfbf8bb65d062 (127.21.28.126:37545): Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
I20260808 06:57:28.713239 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.708944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:40205 (local address 127.0.0.1:39704)
0808 06:57:28.709029 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.709041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:28.709100 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:28.709520 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:28.709521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:28.709524 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:28.709581 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.709583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.710425 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.710426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.712095 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.712098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.713063 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.713064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.713132 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.713138 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 06:57:28.713163 (+ 25us) client_negotiation.cc:240] Negotiation successful
0808 06:57:28.713197 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":48,"threads_started":1}
I20260808 06:57:28.713485 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.709085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.21.28.125:40205)
0808 06:57:28.709354 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:28.709356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:28.709362 (+ 6us) server_negotiation.cc:407] Connection header received
0808 06:57:28.709384 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:28.709385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:28.709394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:28.709636 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:28.709667 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.710332 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.712183 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.712968 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.713364 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.713378 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:28.713396 (+ 18us) server_negotiation.cc:299] Negotiation successful
0808 06:57:28.713442 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":66,"threads_started":1}
I20260808 06:57:28.713707 31349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35b386fafcdc4068a44c02ef082a8246" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cffb51c0521e4dec9fb886b67223f949" is_pre_election: true
I20260808 06:57:28.713793 31349 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35b386fafcdc4068a44c02ef082a8246 in term 1.
I20260808 06:57:28.713953 31584 leader_election.cc:302] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35b386fafcdc4068a44c02ef082a8246, cffb51c0521e4dec9fb886b67223f949; no voters: e1fa7e28f9544dd1a57bfbf8bb65d062
I20260808 06:57:28.714054 32402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 06:57:28.714129 32402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1fa7e28f9544dd1a57bfbf8bb65d062)
I20260808 06:57:28.714147 32406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.708781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:39025 (local address 127.0.0.1:41524)
0808 06:57:28.709124 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.709135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:28.709189 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:28.709501 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:28.709503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:28.709506 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:28.709566 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.709568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.710688 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.710689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.712383 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.712387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.713929 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.713930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.713997 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.714000 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:28.714044 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 06:57:28.714083 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":47,"threads_started":1}
I20260808 06:57:28.714160 32402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:57:28.714320 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.708899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.21.28.124:39025)
0808 06:57:28.709261 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:28.709263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:28.709272 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:28.709329 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:28.709330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:28.709341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:28.709426 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:28.709640 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.710588 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.712621 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.713763 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.713824 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.713836 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:28.714231 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 06:57:28.714269 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":39,"threads_started":1}
I20260808 06:57:28.714509 31617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cffb51c0521e4dec9fb886b67223f949" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "35b386fafcdc4068a44c02ef082a8246" is_pre_election: true
I20260808 06:57:28.714993 32402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } }
I20260808 06:57:28.715111 32402 leader_election.cc:288] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [CANDIDATE]: Term 2 election: Requested vote from peers e1fa7e28f9544dd1a57bfbf8bb65d062 (127.21.28.126:37545), cffb51c0521e4dec9fb886b67223f949 (127.21.28.125:40205)
I20260808 06:57:28.715113 31617 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cffb51c0521e4dec9fb886b67223f949 in current term 2: Already voted for candidate 35b386fafcdc4068a44c02ef082a8246 in this term.
I20260808 06:57:28.715276 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.715175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46544)
0808 06:57:28.715210 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.715235 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260808 06:57:28.715349 31349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35b386fafcdc4068a44c02ef082a8246" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cffb51c0521e4dec9fb886b67223f949"
I20260808 06:57:28.715384 31315 leader_election.cc:302] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cffb51c0521e4dec9fb886b67223f949; no voters: 35b386fafcdc4068a44c02ef082a8246, e1fa7e28f9544dd1a57bfbf8bb65d062
I20260808 06:57:28.715441 31349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:57:28.715742 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.715625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46546)
0808 06:57:28.715696 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.715720 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 06:57:28.715858 31583 leader_election.cc:334] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1fa7e28f9544dd1a57bfbf8bb65d062 (127.21.28.126:37545): Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
I20260808 06:57:28.716275 31349 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35b386fafcdc4068a44c02ef082a8246 in term 2.
I20260808 06:57:28.716353 32401 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:57:28.716468 31584 leader_election.cc:302] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35b386fafcdc4068a44c02ef082a8246, cffb51c0521e4dec9fb886b67223f949; no voters: e1fa7e28f9544dd1a57bfbf8bb65d062
I20260808 06:57:28.716553 32402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:57:28.716674 32402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [term 2 LEADER]: Becoming Leader. State: Replica: 35b386fafcdc4068a44c02ef082a8246, State: Running, Role: LEADER
I20260808 06:57:28.716738 32402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } }
I20260808 06:57:28.717054 32415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35b386fafcdc4068a44c02ef082a8246. Latest consensus state: current_term: 2 leader_uuid: "35b386fafcdc4068a44c02ef082a8246" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:28.717139 32415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:28.717308 32420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:28.717823 32420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:28.717940 32420 catalog_manager.cc:1259] Loaded cluster ID: 28a6f53c4a714db3aa2fb5b40ce56110
I20260808 06:57:28.717971 32420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:28.718115 32420 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:28.718179 32420 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Loaded TSK: 0
I20260808 06:57:28.718317 32420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:57:28.766585 31603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:28.768347 31603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:28.769332 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.769231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46558)
0808 06:57:28.769273 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.769301 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 06:57:28.769372 31349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [term 2 FOLLOWER]: Refusing update from remote peer 35b386fafcdc4068a44c02ef082a8246: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 06:57:28.769665 32415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 06:57:28.769759 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.769672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.0.0.1:46570)
0808 06:57:28.769710 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.769734 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 06:57:28.769825 31583 consensus_peers.cc:487] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 -> Peer e1fa7e28f9544dd1a57bfbf8bb65d062 (127.21.28.126:37545): Couldn't send request to peer e1fa7e28f9544dd1a57bfbf8bb65d062. Status: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 06:57:28.770278 32401 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35b386fafcdc4068a44c02ef082a8246. Latest consensus state: current_term: 2 leader_uuid: "35b386fafcdc4068a44c02ef082a8246" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:28.770367 32401 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:57:28.770793 32402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "35b386fafcdc4068a44c02ef082a8246" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:28.770891 32402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:28.770917 32401 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "35b386fafcdc4068a44c02ef082a8246" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1fa7e28f9544dd1a57bfbf8bb65d062" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 37545 } } peers { permanent_uuid: "cffb51c0521e4dec9fb886b67223f949" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 40205 } attrs { promote: false } } peers { permanent_uuid: "35b386fafcdc4068a44c02ef082a8246" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 39025 } attrs { promote: false } } }
I20260808 06:57:28.770992 32401 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb51c0521e4dec9fb886b67223f949 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:57:28.771107 32424 mvcc.cc:205] Tried to move back new op lower bound from 7316161530957496320 to 7316161530744307712. Current Snapshot: MvccSnapshot[applied={T|T < 7316161530957496320}]
I20260808 06:57:28.775779 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.772666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.21.28.65:38769)
0808 06:57:28.772784 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:28.772786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:28.772793 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:28.772818 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:28.772819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:28.772829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:28.773046 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:28.773084 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.773664 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.775059 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.775440 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.775501 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.775513 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:28.775642 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 06:57:28.775703 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260808 06:57:28.775769 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.772531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:38769 (local address 127.0.0.1:58494)
0808 06:57:28.772636 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.772646 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:28.772682 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:28.772911 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:28.772912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:28.772913 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:28.772973 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.772975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.773734 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.773735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.774964 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.774966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.775513 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.775514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.775578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.775583 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:57:28.775630 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 06:57:28.775692 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":60,"thread_start_us":73,"threads_started":1}
I20260808 06:57:28.775863 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.772445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:33083 (local address 127.0.0.1:58804)
0808 06:57:28.772498 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.772507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:28.772551 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:28.772826 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:28.772827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:28.772830 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:28.772884 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.772886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.773773 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.773774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.774964 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.774966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.775696 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.775697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.775749 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.775752 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:28.775779 (+ 27us) client_negotiation.cc:240] Negotiation successful
0808 06:57:28.775811 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":37}
I20260808 06:57:28.775918 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.772540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.21.28.66:46533)
0808 06:57:28.772900 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:28.772902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:28.772909 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:28.772931 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:28.772932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:28.772945 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:28.772999 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:28.773152 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.773918 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.774892 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.775465 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.775527 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.775540 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:28.775801 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 06:57:28.775842 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":51,"threads_started":1}
I20260808 06:57:28.775985 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.772459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46533 (local address 127.0.0.1:45784)
0808 06:57:28.772641 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.772648 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:28.772684 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:28.773043 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:28.773044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:28.773046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:28.773078 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.773079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.773981 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.773982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.774810 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.774812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.775573 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.775574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.775632 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.775636 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:28.775703 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 06:57:28.775742 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260808 06:57:28.776001 31776 tablet_service.cc:1467] Processing CreateTablet for tablet cb9ddf7d445e413f992cb7c7d35cc029 (DEFAULT_TABLE table=test-table [id=0539dec83f68492699f180931a0c8b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:28.776135 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9ddf7d445e413f992cb7c7d35cc029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:28.776567 31983 tablet_service.cc:1467] Processing CreateTablet for tablet cb9ddf7d445e413f992cb7c7d35cc029 (DEFAULT_TABLE table=test-table [id=0539dec83f68492699f180931a0c8b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:28.776671 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.772499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.21.28.67:33083)
0808 06:57:28.772678 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:28.772680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:28.772688 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:28.772714 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:28.772715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:28.772726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:28.772816 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:28.772974 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.773711 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.775042 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.775593 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.776520 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.776537 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:28.776555 (+ 18us) server_negotiation.cc:299] Negotiation successful
0808 06:57:28.776603 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260808 06:57:28.776705 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9ddf7d445e413f992cb7c7d35cc029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:28.777367 32435 tablet_bootstrap.cc:492] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b: Bootstrap starting.
I20260808 06:57:28.777498 32198 tablet_service.cc:1467] Processing CreateTablet for tablet cb9ddf7d445e413f992cb7c7d35cc029 (DEFAULT_TABLE table=test-table [id=0539dec83f68492699f180931a0c8b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:28.777601 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9ddf7d445e413f992cb7c7d35cc029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:28.777758 32435 tablet_bootstrap.cc:654] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:28.778049 32436 tablet_bootstrap.cc:492] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: Bootstrap starting.
I20260808 06:57:28.778328 32436 tablet_bootstrap.cc:654] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:28.778687 32445 tablet_bootstrap.cc:492] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: Bootstrap starting.
I20260808 06:57:28.778837 32436 tablet_bootstrap.cc:492] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: No bootstrap required, opened a new log
I20260808 06:57:28.778887 32436 ts_tablet_manager.cc:1397] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:28.778968 32435 tablet_bootstrap.cc:492] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b: No bootstrap required, opened a new log
I20260808 06:57:28.779018 32435 ts_tablet_manager.cc:1397] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:28.779021 32436 raft_consensus.cc:348] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.779093 32436 raft_consensus.cc:374] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:28.779095 32445 tablet_bootstrap.cc:654] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:28.779114 32436 raft_consensus.cc:729] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e96629b8965148f98b194c92e1997d6a, State: Initialized, Role: FOLLOWER
I20260808 06:57:28.779132 32435 raft_consensus.cc:348] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.779155 32436 consensus_queue.cc:260] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.779192 32435 raft_consensus.cc:374] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:28.779213 32435 raft_consensus.cc:729] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6350088346744faacf9843062dd1e2b, State: Initialized, Role: FOLLOWER
I20260808 06:57:28.779248 32435 consensus_queue.cc:260] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.779304 32436 ts_tablet_manager.cc:1428] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:28.779417 32435 ts_tablet_manager.cc:1428] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:28.779893 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.779659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.65:35483)
0808 06:57:28.779816 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.779854 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":113,"threads_started":1}
I20260808 06:57:28.780050 31884 heartbeater.cc:503] Master 127.21.28.124:39025 was elected leader, sending a full tablet report...
I20260808 06:57:28.780356 32445 tablet_bootstrap.cc:492] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: No bootstrap required, opened a new log
I20260808 06:57:28.780412 32445 ts_tablet_manager.cc:1397] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:28.780388 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.780194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.66:57817)
0808 06:57:28.780317 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.780355 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260808 06:57:28.780521 32445 raft_consensus.cc:348] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.780432 32091 heartbeater.cc:503] Master 127.21.28.124:39025 was elected leader, sending a full tablet report...
I20260808 06:57:28.780597 32445 raft_consensus.cc:374] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:28.780623 32445 raft_consensus.cc:729] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4792a68b34154ac9872393a4934b71d1, State: Initialized, Role: FOLLOWER
I20260808 06:57:28.780659 32445 consensus_queue.cc:260] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.780800 32445 ts_tablet_manager.cc:1428] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:28.781158 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.781052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.66:39103)
0808 06:57:28.781103 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.781132 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 06:57:28.781344 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.781000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.67:48009)
0808 06:57:28.781275 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.781311 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":70,"threads_started":1}
I20260808 06:57:28.781378 32300 heartbeater.cc:503] Master 127.21.28.124:39025 was elected leader, sending a full tablet report...
I20260808 06:57:28.781838 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.781601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.67:35851)
0808 06:57:28.781631 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.781810 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260808 06:57:28.848239 32447 raft_consensus.cc:482] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:28.848325 32447 raft_consensus.cc:504] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.848644 32447 leader_election.cc:288] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4792a68b34154ac9872393a4934b71d1 (127.21.28.67:33083), c6350088346744faacf9843062dd1e2b (127.21.28.65:38769)
I20260808 06:57:28.851485 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.849028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:34101 (local address 127.21.28.65:38769)
0808 06:57:28.849079 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:28.849081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:28.849141 (+ 60us) server_negotiation.cc:407] Connection header received
0808 06:57:28.849191 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:28.849192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:28.849201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:28.849248 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:28.849377 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.849862 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.850835 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.851193 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.851252 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.851262 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:28.851385 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 06:57:28.851418 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 06:57:28.851529 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.848891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:38769 (local address 127.21.28.66:34101)
0808 06:57:28.849085 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.849093 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:28.849141 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:28.849257 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:28.849258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:28.849259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:28.849324 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.849326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.849934 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.849935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.850756 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.850758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.851262 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.851264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.851327 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.851331 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:28.851374 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 06:57:28.851418 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":16,"thread_start_us":124,"threads_started":1}
I20260808 06:57:28.851542 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.848794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:33083 (local address 127.21.28.66:46559)
0808 06:57:28.848844 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.848856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:28.848943 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:28.849155 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:28.849157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:28.849159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:28.849248 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.849250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.849888 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.849889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.850738 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:28.850740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.851285 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.851286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:28.851364 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.851366 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:28.851410 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 06:57:28.851449 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":37}
I20260808 06:57:28.851730 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.848889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:46559 (local address 127.21.28.67:33083)
0808 06:57:28.848940 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:28.848942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:28.848952 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:28.849049 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:28.849050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:28.849062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:28.849121 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:28.849315 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.849818 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.850829 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.851192 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.851258 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.851268 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:28.851429 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 06:57:28.851470 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 06:57:28.851817 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9ddf7d445e413f992cb7c7d35cc029" candidate_uuid: "e96629b8965148f98b194c92e1997d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6350088346744faacf9843062dd1e2b" is_pre_election: true
I20260808 06:57:28.851903 31804 raft_consensus.cc:2454] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e96629b8965148f98b194c92e1997d6a in term 0.
I20260808 06:57:28.852056 31921 leader_election.cc:302] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6350088346744faacf9843062dd1e2b, e96629b8965148f98b194c92e1997d6a; no voters:
I20260808 06:57:28.852097 32229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9ddf7d445e413f992cb7c7d35cc029" candidate_uuid: "e96629b8965148f98b194c92e1997d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4792a68b34154ac9872393a4934b71d1" is_pre_election: true
I20260808 06:57:28.852190 32447 raft_consensus.cc:2781] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:28.852232 32447 raft_consensus.cc:482] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:28.852239 32229 raft_consensus.cc:2454] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e96629b8965148f98b194c92e1997d6a in term 0.
I20260808 06:57:28.852258 32447 raft_consensus.cc:3037] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:28.852999 32447 raft_consensus.cc:504] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.853101 32447 leader_election.cc:288] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [CANDIDATE]: Term 1 election: Requested vote from peers 4792a68b34154ac9872393a4934b71d1 (127.21.28.67:33083), c6350088346744faacf9843062dd1e2b (127.21.28.65:38769)
I20260808 06:57:28.853250 32229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9ddf7d445e413f992cb7c7d35cc029" candidate_uuid: "e96629b8965148f98b194c92e1997d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4792a68b34154ac9872393a4934b71d1"
I20260808 06:57:28.853327 32229 raft_consensus.cc:3037] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:28.853314 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9ddf7d445e413f992cb7c7d35cc029" candidate_uuid: "e96629b8965148f98b194c92e1997d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6350088346744faacf9843062dd1e2b"
I20260808 06:57:28.853389 31804 raft_consensus.cc:3037] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:28.853952 31804 raft_consensus.cc:2454] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e96629b8965148f98b194c92e1997d6a in term 1.
I20260808 06:57:28.853962 32229 raft_consensus.cc:2454] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e96629b8965148f98b194c92e1997d6a in term 1.
I20260808 06:57:28.854112 31921 leader_election.cc:302] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6350088346744faacf9843062dd1e2b, e96629b8965148f98b194c92e1997d6a; no voters:
I20260808 06:57:28.854194 32447 raft_consensus.cc:2781] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:28.854332 32447 raft_consensus.cc:686] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 1 LEADER]: Becoming Leader. State: Replica: e96629b8965148f98b194c92e1997d6a, State: Running, Role: LEADER
I20260808 06:57:28.854416 32447 consensus_queue.cc:237] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } }
I20260808 06:57:28.854907 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.854814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.66:51073)
0808 06:57:28.854855 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.854884 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 06:57:28.855115 31604 catalog_manager.cc:5495] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a reported cstate change: term changed from 0 to 1, leader changed from <none> to e96629b8965148f98b194c92e1997d6a (127.21.28.66). New cstate: current_term: 1 leader_uuid: "e96629b8965148f98b194c92e1997d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e96629b8965148f98b194c92e1997d6a" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 } health_report { overall_health: UNKNOWN } } }
I20260808 06:57:28.866673 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.864788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.21.28.66:46533)
0808 06:57:28.864840 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:28.864841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:28.864849 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:28.864878 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:28.864879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:28.864893 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:28.864948 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:28.865043 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.865475 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.866123 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:28.866287 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:28.866326 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:28.866441 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:28.866530 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:28.866576 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:28.866609 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 06:57:28.866622 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 06:57:28.867585 31804 raft_consensus.cc:1261] T cb9ddf7d445e413f992cb7c7d35cc029 P c6350088346744faacf9843062dd1e2b [term 1 FOLLOWER]: Refusing update from remote peer e96629b8965148f98b194c92e1997d6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:28.867689 32229 raft_consensus.cc:1261] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 1 FOLLOWER]: Refusing update from remote peer e96629b8965148f98b194c92e1997d6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:28.867879 32447 consensus_queue.cc:1035] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6350088346744faacf9843062dd1e2b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 38769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:28.868013 32463 consensus_queue.cc:1035] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4792a68b34154ac9872393a4934b71d1" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:28.868813 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.868710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.67:52445)
0808 06:57:28.868754 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.868783 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 06:57:28.869341 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.869227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.65:55813)
0808 06:57:28.869270 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.869308 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260808 06:57:28.870175 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.870070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.66:57737)
0808 06:57:28.870118 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.870148 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 06:57:28.870523 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.870426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.65:37211)
0808 06:57:28.870463 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.870496 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260808 06:57:28.871269 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.871184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:37545 (local address 127.21.28.67:58997)
0808 06:57:28.871213 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.871239 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260808 06:57:28.871956 31603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:28.872043 31603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:28.873328 31603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35b386fafcdc4068a44c02ef082a8246: Sending DeleteTablet for 3 replicas of tablet cb9ddf7d445e413f992cb7c7d35cc029
I20260808 06:57:28.873606 31983 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9ddf7d445e413f992cb7c7d35cc029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:28 UTC) from {username='oryx', principal='oryx/127.21.28.124@KRBTEST.COM'} at 127.0.0.1:45784
I20260808 06:57:28.873787 32479 tablet_replica.cc:330] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: stopping tablet replica
I20260808 06:57:28.873842 32479 raft_consensus.cc:2229] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:28.873953 32479 raft_consensus.cc:2258] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:28.874081 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 31553
I20260808 06:57:28.874244 32479 ts_tablet_manager.cc:1905] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:28.875953 32479 ts_tablet_manager.cc:1918] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:28.876003 32479 log.cc:1196] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/cb9ddf7d445e413f992cb7c7d35cc029
I20260808 06:57:28.876195 32479 ts_tablet_manager.cc:1939] T cb9ddf7d445e413f992cb7c7d35cc029 P e96629b8965148f98b194c92e1997d6a: Deleting consensus metadata
I20260808 06:57:28.881498 31583 catalog_manager.cc:4841] TS e96629b8965148f98b194c92e1997d6a (127.21.28.66:46533): tablet cb9ddf7d445e413f992cb7c7d35cc029 (table test-table [id=0539dec83f68492699f180931a0c8b95]) successfully deleted
I20260808 06:57:28.881735 32198 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9ddf7d445e413f992cb7c7d35cc029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:28 UTC) from {username='oryx', principal='oryx/127.21.28.124@KRBTEST.COM'} at 127.0.0.1:58804
I20260808 06:57:28.883706 32480 tablet_replica.cc:330] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: stopping tablet replica
I20260808 06:57:28.883791 32480 raft_consensus.cc:2229] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:28.883854 32480 raft_consensus.cc:2258] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:28.884199 32480 ts_tablet_manager.cc:1905] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:28.885915 32480 ts_tablet_manager.cc:1918] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:28.885986 32480 log.cc:1196] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/cb9ddf7d445e413f992cb7c7d35cc029
I20260808 06:57:28.886260 32480 ts_tablet_manager.cc:1939] T cb9ddf7d445e413f992cb7c7d35cc029 P 4792a68b34154ac9872393a4934b71d1: Deleting consensus metadata
I20260808 06:57:28.886605 31584 catalog_manager.cc:4841] TS 4792a68b34154ac9872393a4934b71d1 (127.21.28.67:33083): tablet cb9ddf7d445e413f992cb7c7d35cc029 (table test-table [id=0539dec83f68492699f180931a0c8b95]) successfully deleted
W20260808 06:57:28.886751 31584 connection.cc:535] client connection to 127.21.28.65:38769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 06:57:28.887120 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:28.887013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:38769 (local address 127.0.0.1:58500)
0808 06:57:28.887056 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:28.887086 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260808 06:57:28.887274 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 31894
W20260808 06:57:28.887270 31584 catalog_manager.cc:4584] TS c6350088346744faacf9843062dd1e2b (127.21.28.65:38769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb9ddf7d445e413f992cb7c7d35cc029: Network error: Client connection negotiation failed: client connection to 127.21.28.65:38769: connect: Connection refused (error 111)
I20260808 06:57:28.896257 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 32103
I20260808 06:57:28.907114 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 31134
I20260808 06:57:28.916357 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 31410
2026-08-08T06:57:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5219 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:28 dist-test-slave-nj21 krb5kdc[32484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:28 dist-test-slave-nj21 krb5kdc[32484](info): set up 2 sockets
Aug 08 06:57:28 dist-test-slave-nj21 krb5kdc[32484](info): commencing operation
krb5kdc: starting...
W20260808 06:57:30.050271 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.126.keytab.
Entry for principal oryx/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.126.keytab.
I20260808 06:57:30.165863 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:42223
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.126.keytab
--principal=oryx/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:30.245867 32501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:30.246074 32501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:30.246095 32501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:30.247239 32501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:30.247282 32501 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 06:57:30.247293 32501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:30.247318 32501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:30.247340 32501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:30.247362 32501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:30.248890 32501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:42223
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.126.keytab
--principal=oryx/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:42223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:30.249120 32501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:30.249326 32501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:30.253947 32510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:30.253955 32507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:30.253959 32512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:30.254210 32501 server_base.cc:1034] running on GCE node
I20260808 06:57:30.254441 32501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:30.254715 32501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:30.255877 32501 hybrid_clock.cc:648] HybridClock initialized: now 1786172250255858 us; error 30 us; skew 500 ppm
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:30.258191 32501 init.cc:377] Logged in from keytab as oryx/127.21.28.126@KRBTEST.COM (short username oryx)
I20260808 06:57:30.258605 32501 webserver.cc:460] Webserver started at http://127.21.28.126:34257/ using document root <none> and password file <none>
I20260808 06:57:30.258805 32501 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:30.258852 32501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:30.258983 32501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:30.260048 32501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "e5cffb09f87e428a9491fc43d2e6c2e5"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "b5d677c268159c3fd77801591e83d7e9"
server_key_iv: "d49755e87fd6245d49cc3fe447eda5ee"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.260373 32501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "e5cffb09f87e428a9491fc43d2e6c2e5"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "b5d677c268159c3fd77801591e83d7e9"
server_key_iv: "d49755e87fd6245d49cc3fe447eda5ee"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.261596 32501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:30.262262 32521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.262408 32501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:30.262491 32501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "e5cffb09f87e428a9491fc43d2e6c2e5"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "b5d677c268159c3fd77801591e83d7e9"
server_key_iv: "d49755e87fd6245d49cc3fe447eda5ee"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.262548 32501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:30.272014 32501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:30.273622 32501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:30.273747 32501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:30.278169 32501 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:42223
I20260808 06:57:30.278232 32609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:42223 every 8 connection(s)
I20260808 06:57:30.278517 32501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:30.279127 32611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:30.280256 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 32501
I20260808 06:57:30.280333 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:30.280503 21617 external_mini_cluster.cc:1428] Setting key 9ffc5de8423fb615fd522b7334a9fdc3
I20260808 06:57:30.281836 32611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: Bootstrap starting.
I20260808 06:57:30.282495 32611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:30.282804 32611 log.cc:824] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:30.283504 32611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: No bootstrap required, opened a new log
I20260808 06:57:30.285655 32611 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cffb09f87e428a9491fc43d2e6c2e5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42223 } }
I20260808 06:57:30.285789 32611 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:30.285809 32611 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cffb09f87e428a9491fc43d2e6c2e5, State: Initialized, Role: FOLLOWER
I20260808 06:57:30.285920 32611 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cffb09f87e428a9491fc43d2e6c2e5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42223 } }
I20260808 06:57:30.285987 32611 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:30.286011 32611 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:30.286056 32611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:30.286563 32611 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cffb09f87e428a9491fc43d2e6c2e5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42223 } }
I20260808 06:57:30.286671 32611 leader_election.cc:302] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5cffb09f87e428a9491fc43d2e6c2e5; no voters:
I20260808 06:57:30.286820 32611 leader_election.cc:288] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:30.286855 32616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:30.287024 32616 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [term 1 LEADER]: Becoming Leader. State: Replica: e5cffb09f87e428a9491fc43d2e6c2e5, State: Running, Role: LEADER
I20260808 06:57:30.287071 32611 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:30.287181 32616 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cffb09f87e428a9491fc43d2e6c2e5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42223 } }
I20260808 06:57:30.287499 32616 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5cffb09f87e428a9491fc43d2e6c2e5. Latest consensus state: current_term: 1 leader_uuid: "e5cffb09f87e428a9491fc43d2e6c2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cffb09f87e428a9491fc43d2e6c2e5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42223 } } }
I20260808 06:57:30.287575 32616 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:30.287513 32618 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5cffb09f87e428a9491fc43d2e6c2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cffb09f87e428a9491fc43d2e6c2e5" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42223 } } }
I20260808 06:57:30.287632 32618 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:30.287814 32626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:30.288499 32626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:30.290309 32626 catalog_manager.cc:1347] Generated new cluster ID: 48bdc875a1f2435c950dd74af388b00e
I20260808 06:57:30.290359 32626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:30.291451 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.280975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.21.28.126:42223)
0808 06:57:30.281153 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.281162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.281196 (+ 34us) server_negotiation.cc:407] Connection header received
0808 06:57:30.281684 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.281693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.281752 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.281928 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.282506 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.283814 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.284527 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.285144 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.285255 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.287838 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:30.287842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:30.287843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:30.287854 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:30.289655 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.289963 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.289966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.289966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.290067 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.290272 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.290273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.290273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.290555 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:30.290666 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.291123 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.291242 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260808 06:57:30.304148 32626 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:30.304667 32626 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:30.309561 32626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: Generated new TSK 0
I20260808 06:57:30.309712 32626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab.
Entry for principal oryx/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab.
I20260808 06:57:30.337813 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42223
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab
--principal=oryx/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:30.418998 32652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:30.419212 32652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:30.419235 32652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:30.420406 32652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:30.420449 32652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:30.420470 32652 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:30.420501 32652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:30.422049 32652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab
--principal=oryx/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:42223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:30.422291 32652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:30.422502 32652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:57:30.427436 32652 server_base.cc:1034] running on GCE node
W20260808 06:57:30.427459 32664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:30.427505 32659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:30.427562 32658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:30.427878 32652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:30.428092 32652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:30.429230 32652 hybrid_clock.cc:648] HybridClock initialized: now 1786172250429217 us; error 27 us; skew 500 ppm
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:30.431797 32652 init.cc:377] Logged in from keytab as oryx/127.21.28.65@KRBTEST.COM (short username oryx)
I20260808 06:57:30.432204 32652 webserver.cc:460] Webserver started at http://127.21.28.65:40713/ using document root <none> and password file <none>
I20260808 06:57:30.432425 32652 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:30.432474 32652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:30.432598 32652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:30.433631 32652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "9de4d11a44674988b2f2d5ef223b1b6a"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "e69fa0dee998968b727222677189519b"
server_key_iv: "67e4460197fe604db87776ecc3fb6f58"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.433956 32652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "9de4d11a44674988b2f2d5ef223b1b6a"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "e69fa0dee998968b727222677189519b"
server_key_iv: "67e4460197fe604db87776ecc3fb6f58"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.435228 32652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 06:57:30.435942 32677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.436082 32652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.436175 32652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "9de4d11a44674988b2f2d5ef223b1b6a"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "e69fa0dee998968b727222677189519b"
server_key_iv: "67e4460197fe604db87776ecc3fb6f58"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.436232 32652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:30.446084 32652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:30.447801 32652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:30.447940 32652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:30.448436 32652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:30.448475 32652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.448503 32652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:30.448529 32652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:30.455436 32652 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:40003
I20260808 06:57:30.455547 396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:40003 every 8 connection(s)
I20260808 06:57:30.455880 32652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:30.457644 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.448860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:37739 (local address 127.21.28.126:42223)
0808 06:57:30.448934 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.448937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.449098 (+ 161us) server_negotiation.cc:407] Connection header received
0808 06:57:30.449573 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.449575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.449589 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.449673 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.450419 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.451173 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.452348 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.452561 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.452622 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.455019 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:30.455021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:30.455021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:30.455028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:30.456574 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.456907 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.456908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.456908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.456957 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.457116 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.457116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.457117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.457213 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:30.457258 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.457476 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.457575 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 06:57:30.457741 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.448816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42223 (local address 127.21.28.65:37739)
0808 06:57:30.448970 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.449010 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.449488 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.449723 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.449726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.450035 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:30.450317 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.450323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.451292 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.451293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.452255 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.452258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.452676 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.452677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.452790 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.453242 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:30.453250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:30.454935 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:30.456641 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.456645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.456647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.456850 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.456993 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.456994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.456994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.457077 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.457256 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:30.457259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:30.457365 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.457451 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.457580 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260808 06:57:30.458264 399 heartbeater.cc:347] Connected to a master server at 127.21.28.126:42223
I20260808 06:57:30.458354 399 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:30.458532 399 heartbeater.cc:511] Master 127.21.28.126:42223 requested a full tablet report, sending...
I20260808 06:57:30.459059 32541 ts_manager.cc:194] Registered new tserver with Master: 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003)
I20260808 06:57:30.459825 32541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:37739
I20260808 06:57:30.462817 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 32652
I20260808 06:57:30.462920 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:30.463116 21617 external_mini_cluster.cc:1428] Setting key ccb58af4c3b2bca15858084d5ba37bb1
I20260808 06:57:30.464357 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.459345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:34893 (local address 127.21.28.126:42223)
0808 06:57:30.459406 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.459408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.459416 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:30.459434 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.459435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.459444 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.459490 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.459826 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.460447 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.461110 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.461285 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.461336 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.462109 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:30.462110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:30.462110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:30.462114 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:30.463292 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.463648 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.463649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.463649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.463726 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.463883 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.463884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.463885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.463993 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:30.464049 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.464194 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.464266 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 06:57:30.464367 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.459196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42223 (local address 127.21.28.65:34893)
0808 06:57:30.459323 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.459333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.459391 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.459492 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.459493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.459658 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:30.459759 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.459760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.460551 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.460551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.461050 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.461051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.461338 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.461338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.461379 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.461748 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:30.461753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:30.462053 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:30.463432 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.463432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.463433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.463586 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.463780 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.463781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.463781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.463835 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.464049 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:30.464050 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:30.464095 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.464164 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.464265 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab.
Entry for principal oryx/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab.
I20260808 06:57:30.486580 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42223
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab
--principal=oryx/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:30.567741 411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:30.567943 411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:30.567976 411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:30.569114 411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:30.569155 411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:30.569183 411 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:30.569206 411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:30.570720 411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab
--principal=oryx/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:42223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:30.570948 411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:30.571151 411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:30.575843 420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:30.575835 421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:30.575935 411 server_base.cc:1034] running on GCE node
W20260808 06:57:30.575834 423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:30.576188 411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:30.576376 411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:30.577512 411 hybrid_clock.cc:648] HybridClock initialized: now 1786172250577498 us; error 28 us; skew 500 ppm
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:30.579896 411 init.cc:377] Logged in from keytab as oryx/127.21.28.66@KRBTEST.COM (short username oryx)
I20260808 06:57:30.580293 411 webserver.cc:460] Webserver started at http://127.21.28.66:38601/ using document root <none> and password file <none>
I20260808 06:57:30.580503 411 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:30.580549 411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:30.580677 411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:30.581652 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "7e086e9a1d1d4791842bc5464e70e61b"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "e048d73a4bede817ceac755759640fe0"
server_key_iv: "04cb74e2bd50084310627ee625f213ab"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.581986 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "7e086e9a1d1d4791842bc5464e70e61b"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "e048d73a4bede817ceac755759640fe0"
server_key_iv: "04cb74e2bd50084310627ee625f213ab"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.583220 411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:30.583882 435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.584019 411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.584093 411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "7e086e9a1d1d4791842bc5464e70e61b"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "e048d73a4bede817ceac755759640fe0"
server_key_iv: "04cb74e2bd50084310627ee625f213ab"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.584151 411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:30.594033 411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:30.595616 411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:30.595772 411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:30.596295 411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:30.596336 411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.596364 411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:30.596390 411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:30.603260 411 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:46341
I20260808 06:57:30.603538 623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:46341 every 8 connection(s)
I20260808 06:57:30.603582 411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:30.605322 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.596633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:33185 (local address 127.21.28.126:42223)
0808 06:57:30.596702 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.596704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.596895 (+ 191us) server_negotiation.cc:407] Connection header received
0808 06:57:30.597347 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.597349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.597361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.597442 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.598019 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.598774 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.600072 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.600276 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.600334 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.602609 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:30.602610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:30.602611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:30.602619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:30.604046 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.604462 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.604464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.604464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.604521 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.604727 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.604728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.604729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.604842 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:30.604896 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.605147 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.605233 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 06:57:30.605466 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.596616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42223 (local address 127.21.28.66:33185)
0808 06:57:30.596778 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.596817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.597264 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.597490 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.597492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.597719 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:30.597946 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.597951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.598871 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.598872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.599991 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.599994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.600357 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.600359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.600478 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.600943 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:30.600952 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:30.602525 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:30.604120 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.604127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.604129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.604394 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.604571 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.604572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.604572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.604673 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.604894 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:30.604900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:30.605032 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.605117 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.605267 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260808 06:57:30.605860 625 heartbeater.cc:347] Connected to a master server at 127.21.28.126:42223
I20260808 06:57:30.605940 625 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:30.606112 625 heartbeater.cc:511] Master 127.21.28.126:42223 requested a full tablet report, sending...
I20260808 06:57:30.606518 32542 ts_manager.cc:194] Registered new tserver with Master: 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341)
I20260808 06:57:30.606992 32542 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:33185
I20260808 06:57:30.611546 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.606811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:52281 (local address 127.21.28.126:42223)
0808 06:57:30.606845 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.606846 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.607016 (+ 170us) server_negotiation.cc:407] Connection header received
0808 06:57:30.607071 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.607072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.607081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.607124 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.607413 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.607949 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.608607 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.608785 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.608837 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.609567 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:30.609569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:30.609569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:30.609575 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:30.610541 (+ 966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.610854 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.610855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.610855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.610904 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.611061 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.611062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.611062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.611191 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:30.611255 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.611390 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.611460 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 06:57:30.611559 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.606775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42223 (local address 127.21.28.66:52281)
0808 06:57:30.606956 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.606965 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.607019 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.607126 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.607127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.607289 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:30.607347 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.607349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.608048 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.608049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.608541 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.608543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.608843 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.608844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.608904 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.609233 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:30.609237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:30.609494 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:30.610613 (+ 1119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.610613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.610614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.610793 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.610956 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.610957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.610957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.611012 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.611258 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:30.611259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:30.611303 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.611362 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.611456 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260808 06:57:30.611595 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 411
I20260808 06:57:30.611733 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:30.611963 21617 external_mini_cluster.cc:1428] Setting key ca62fd1061c7c23de4865f7d734e25ca
WARNING: no policy specified for oryx/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab.
Entry for principal oryx/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab.
I20260808 06:57:30.638001 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:42223
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab
--principal=oryx/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:30.719956 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:30.720153 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:30.720175 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:30.721251 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:30.721283 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:30.721302 635 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:57:30.721323 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:30.722793 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab
--principal=oryx/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.28.126:42223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:30.722980 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:30.723165 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:30.728000 643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:30.727980 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:30.728006 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:30.728041 635 server_base.cc:1034] running on GCE node
I20260808 06:57:30.728428 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:30.728667 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:30.729801 635 hybrid_clock.cc:648] HybridClock initialized: now 1786172250729789 us; error 28 us; skew 500 ppm
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:30.732226 635 init.cc:377] Logged in from keytab as oryx/127.21.28.67@KRBTEST.COM (short username oryx)
I20260808 06:57:30.732672 635 webserver.cc:460] Webserver started at http://127.21.28.67:41551/ using document root <none> and password file <none>
I20260808 06:57:30.732865 635 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:30.732916 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:30.733037 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:30.734037 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "a1f63b79076f480eb77a58c04b2ed08e"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "6a58d663349525c4580cc658860f8d55"
server_key_iv: "19f0b866035cb0d6a8f64fcb1047878b"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.734359 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "a1f63b79076f480eb77a58c04b2ed08e"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "6a58d663349525c4580cc658860f8d55"
server_key_iv: "19f0b866035cb0d6a8f64fcb1047878b"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.735587 635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 06:57:30.736310 657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.736445 635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:30.736526 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "a1f63b79076f480eb77a58c04b2ed08e"
format_stamp: "Formatted at 2026-08-08 06:57:30 on dist-test-slave-nj21"
server_key: "6a58d663349525c4580cc658860f8d55"
server_key_iv: "19f0b866035cb0d6a8f64fcb1047878b"
server_key_version: "encryptionkey@0"
I20260808 06:57:30.736590 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:30.746256 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:30.747975 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:30.748101 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:30.748608 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:30.748649 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:30.748669 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:30.748682 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:30.755664 635 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:41887
I20260808 06:57:30.755757 822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:41887 every 8 connection(s)
I20260808 06:57:30.756038 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:30.758260 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.748983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:38117 (local address 127.21.28.126:42223)
0808 06:57:30.749035 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.749037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.749225 (+ 188us) server_negotiation.cc:407] Connection header received
0808 06:57:30.749722 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.749724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.749738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.749816 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.750427 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.750966 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.752184 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.752458 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.752516 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.755601 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:30.755603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:30.755603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:30.755611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:30.757043 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.757392 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.757393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.757393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.757468 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.757675 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.757676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.757676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.757791 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:30.757847 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.758093 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.758169 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 06:57:30.758437 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.748980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42223 (local address 127.21.28.67:38117)
0808 06:57:30.749114 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.749147 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.749621 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.749873 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.749876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.750121 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:30.750340 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.750346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.751048 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.751049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.752104 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.752107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.752524 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.752526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.752666 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.753187 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:30.753197 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:30.755504 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:30.757113 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.757118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.757120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.757329 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.757519 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.757520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.757520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.757622 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.757843 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:30.757849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:30.757976 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.758061 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.758210 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260808 06:57:30.758894 823 heartbeater.cc:347] Connected to a master server at 127.21.28.126:42223
I20260808 06:57:30.758980 823 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:30.759142 823 heartbeater.cc:511] Master 127.21.28.126:42223 requested a full tablet report, sending...
I20260808 06:57:30.759507 32541 ts_manager.cc:194] Registered new tserver with Master: a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887)
I20260808 06:57:30.760015 32541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:38117
I20260808 06:57:30.763005 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 635
I20260808 06:57:30.763123 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:30.763332 21617 external_mini_cluster.cc:1428] Setting key 4072fc491ebf0fee7226ec72ac25a77f
I20260808 06:57:30.764142 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:57:30.765202 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.759907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:50691 (local address 127.21.28.126:42223)
0808 06:57:30.759945 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.759947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.759976 (+ 29us) server_negotiation.cc:407] Connection header received
0808 06:57:30.760038 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.760039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.760046 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.760093 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.760357 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.760982 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.761594 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.761773 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.761814 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.762736 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:30.762737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:30.762737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:30.762742 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:30.764198 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.764549 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.764549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.764550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.764603 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.764776 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.764776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.764776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.764863 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:30.764918 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.765055 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.765118 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:57:30.765206 837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.759810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42223 (local address 127.21.28.67:50691)
0808 06:57:30.759923 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.759933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.759990 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.760097 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.760098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.760251 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:30.760302 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.760303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.761054 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.761054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.761542 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.761543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.761947 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.761948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.762007 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.762395 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:30.762399 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:30.762660 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:30.764280 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.764281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.764282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.764492 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.764663 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:30.764664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:30.764664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:30.764722 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:30.764914 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:30.764914 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:30.764959 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.765027 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.765114 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:57:30.785375 32614 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:30.780331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.21.28.126:42223)
0808 06:57:30.780372 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.780374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.780411 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:57:30.780491 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.780492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.780501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.780540 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.781086 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.781554 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.782259 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.782443 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.782494 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.785298 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60890: BlockingRecv error: recv got EOF from 127.0.0.1:60890 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 08 06:57:30 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:30.791994 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.786084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.21.28.126:42223)
0808 06:57:30.786142 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.786144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.786198 (+ 54us) server_negotiation.cc:407] Connection header received
0808 06:57:30.786237 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.786238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.786245 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.786289 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:30.786698 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.787214 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.787925 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.788098 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.788149 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.790138 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:30.790139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:30.790140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:30.790145 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:30.791148 (+ 1003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.791431 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.791432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.791432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.791476 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:30.791601 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:30.791602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:30.791602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:30.791717 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:30.791799 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.791870 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.791928 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 06:57:30.793354 32541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:30.794898 32541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:30.800966 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.796915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:41887 (local address 127.0.0.1:39918)
0808 06:57:30.797177 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.797215 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.797285 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.797592 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.797595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.797598 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:30.797675 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.797677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.798954 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.798954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.800039 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.800041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.800675 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.800676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.800752 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.800765 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.800835 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.800888 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":24,"thread_start_us":62,"threads_started":1}
I20260808 06:57:30.800987 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.797180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.21.28.67:41887)
0808 06:57:30.797387 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.797393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.797404 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:57:30.797456 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.797459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.797474 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.797552 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:30.797787 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.798870 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.800110 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.800599 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.800665 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.800749 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.800851 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.800899 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
I20260808 06:57:30.801136 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.797191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.21.28.65:40003)
0808 06:57:30.797401 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.797405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.797581 (+ 176us) server_negotiation.cc:407] Connection header received
0808 06:57:30.797625 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.797627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.797647 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.797714 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:30.797929 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.798950 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.799935 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.800664 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.800732 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.800804 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.801001 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.801040 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260808 06:57:30.801077 860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.796922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46341 (local address 127.0.0.1:38500)
0808 06:57:30.797375 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.797383 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.797418 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.797692 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.797693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.797695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:30.797736 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.797737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.798614 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.798614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.799834 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.799837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.800840 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.800841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.800918 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.800921 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.800961 (+ 40us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.801010 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260808 06:57:30.801376 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.797190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.21.28.66:46341)
0808 06:57:30.797465 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.797468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.797476 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:30.797507 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.797509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.797521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.797618 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:30.797807 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.798535 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.799930 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.800667 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.800732 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.800804 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.801261 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.801300 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":81,"threads_started":1}
I20260808 06:57:30.801206 859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.796927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40003 (local address 127.0.0.1:60000)
0808 06:57:30.797474 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.797482 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.797523 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.797786 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.797787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.797788 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:30.797844 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.797845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.799004 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.799005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.799852 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.799854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.800841 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.800842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.800907 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.800911 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.800959 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.801153 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":158,"thread_start_us":59,"threads_started":1}
I20260808 06:57:30.802037 32751 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7a1595bcbb41989e2ed0e8af74589c (DEFAULT_TABLE table=test-table [id=4d438e982c6b4b7f95d4860be052a845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:30.802033 738 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7a1595bcbb41989e2ed0e8af74589c (DEFAULT_TABLE table=test-table [id=4d438e982c6b4b7f95d4860be052a845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:30.802134 521 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7a1595bcbb41989e2ed0e8af74589c (DEFAULT_TABLE table=test-table [id=4d438e982c6b4b7f95d4860be052a845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:30.802405 32751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7a1595bcbb41989e2ed0e8af74589c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:30.802405 738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7a1595bcbb41989e2ed0e8af74589c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:30.802475 521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7a1595bcbb41989e2ed0e8af74589c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:30.805641 866 tablet_bootstrap.cc:492] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Bootstrap starting.
I20260808 06:57:30.806210 868 tablet_bootstrap.cc:492] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Bootstrap starting.
I20260808 06:57:30.806330 866 tablet_bootstrap.cc:654] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:30.806699 866 log.cc:824] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:30.806890 868 tablet_bootstrap.cc:654] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:30.807193 868 log.cc:824] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:30.807480 866 tablet_bootstrap.cc:492] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: No bootstrap required, opened a new log
I20260808 06:57:30.807554 866 ts_tablet_manager.cc:1397] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:30.807538 872 tablet_bootstrap.cc:492] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Bootstrap starting.
I20260808 06:57:30.807899 868 tablet_bootstrap.cc:492] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: No bootstrap required, opened a new log
I20260808 06:57:30.807966 868 ts_tablet_manager.cc:1397] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:30.808193 872 tablet_bootstrap.cc:654] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:30.808514 872 log.cc:824] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:30.809286 872 tablet_bootstrap.cc:492] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: No bootstrap required, opened a new log
I20260808 06:57:30.809360 872 ts_tablet_manager.cc:1397] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:30.809501 868 raft_consensus.cc:348] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.809515 866 raft_consensus.cc:348] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.809628 868 raft_consensus.cc:374] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:30.809660 868 raft_consensus.cc:729] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f63b79076f480eb77a58c04b2ed08e, State: Initialized, Role: FOLLOWER
I20260808 06:57:30.809660 866 raft_consensus.cc:374] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:30.809695 866 raft_consensus.cc:729] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e086e9a1d1d4791842bc5464e70e61b, State: Initialized, Role: FOLLOWER
I20260808 06:57:30.809792 868 consensus_queue.cc:260] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.809834 866 consensus_queue.cc:260] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.810118 866 ts_tablet_manager.cc:1428] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 06:57:30.810158 625 heartbeater.cc:503] Master 127.21.28.126:42223 was elected leader, sending a full tablet report...
I20260808 06:57:30.810447 868 ts_tablet_manager.cc:1428] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:30.810493 823 heartbeater.cc:503] Master 127.21.28.126:42223 was elected leader, sending a full tablet report...
I20260808 06:57:30.811261 872 raft_consensus.cc:348] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.811425 872 raft_consensus.cc:374] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:30.811473 872 raft_consensus.cc:729] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de4d11a44674988b2f2d5ef223b1b6a, State: Initialized, Role: FOLLOWER
I20260808 06:57:30.811581 872 consensus_queue.cc:260] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.811820 872 ts_tablet_manager.cc:1428] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260808 06:57:30.811873 399 heartbeater.cc:503] Master 127.21.28.126:42223 was elected leader, sending a full tablet report...
W20260808 06:57:30.854511 626 tablet.cc:2369] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:57:30.956591 401 tablet.cc:2369] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:30.977485 882 raft_consensus.cc:482] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:30.977630 882 raft_consensus.cc:504] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.978319 882 leader_election.cc:288] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341), 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003)
I20260808 06:57:30.980955 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.978340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:42991 (local address 127.21.28.66:46341)
0808 06:57:30.978437 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.978439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.978563 (+ 124us) server_negotiation.cc:407] Connection header received
0808 06:57:30.978624 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.978625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.978635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.978690 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:30.978828 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.979275 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.980252 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.980621 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.980684 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.980693 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.980839 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.980883 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 06:57:30.980973 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.978270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40003 (local address 127.21.28.67:33437)
0808 06:57:30.978391 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.978402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.978450 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.978613 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.978614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.978616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:30.978696 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.978698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.979341 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.979342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.980181 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.980183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.980688 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.980689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.980768 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.980771 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.980821 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.980870 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":41}
I20260808 06:57:30.980994 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.978273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46341 (local address 127.21.28.67:42991)
0808 06:57:30.978496 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:30.978506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:30.978573 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:30.978693 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:30.978693 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:30.978695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:30.978768 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.978769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.979342 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.979343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.980164 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:30.980166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.980692 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.980693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:30.980755 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.980759 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:30.980823 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 06:57:30.980867 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":17,"thread_start_us":92,"threads_started":1}
I20260808 06:57:30.981271 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:30.978380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:33437 (local address 127.21.28.65:40003)
0808 06:57:30.978445 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:30.978447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:30.978457 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:30.978528 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:30.978529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:30.978538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:30.978593 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:30.978761 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.979262 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.980266 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:30.980621 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:30.980681 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:30.980691 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:30.981162 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 06:57:30.981209 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:57:30.981416 551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7a1595bcbb41989e2ed0e8af74589c" candidate_uuid: "a1f63b79076f480eb77a58c04b2ed08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e086e9a1d1d4791842bc5464e70e61b" is_pre_election: true
I20260808 06:57:30.981510 314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7a1595bcbb41989e2ed0e8af74589c" candidate_uuid: "a1f63b79076f480eb77a58c04b2ed08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" is_pre_election: true
I20260808 06:57:30.981587 551 raft_consensus.cc:2454] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1f63b79076f480eb77a58c04b2ed08e in term 0.
I20260808 06:57:30.981645 314 raft_consensus.cc:2454] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1f63b79076f480eb77a58c04b2ed08e in term 0.
I20260808 06:57:30.981801 659 leader_election.cc:302] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e086e9a1d1d4791842bc5464e70e61b, a1f63b79076f480eb77a58c04b2ed08e; no voters:
I20260808 06:57:30.981925 882 raft_consensus.cc:2781] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:30.981978 882 raft_consensus.cc:482] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:30.981994 882 raft_consensus.cc:3037] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:30.982986 882 raft_consensus.cc:504] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.983104 882 leader_election.cc:288] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [CANDIDATE]: Term 1 election: Requested vote from peers 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341), 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003)
I20260808 06:57:30.983237 314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7a1595bcbb41989e2ed0e8af74589c" candidate_uuid: "a1f63b79076f480eb77a58c04b2ed08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de4d11a44674988b2f2d5ef223b1b6a"
I20260808 06:57:30.983309 314 raft_consensus.cc:3037] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:30.983294 551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7a1595bcbb41989e2ed0e8af74589c" candidate_uuid: "a1f63b79076f480eb77a58c04b2ed08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e086e9a1d1d4791842bc5464e70e61b"
I20260808 06:57:30.983379 551 raft_consensus.cc:3037] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:30.984014 314 raft_consensus.cc:2454] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1f63b79076f480eb77a58c04b2ed08e in term 1.
I20260808 06:57:30.984157 551 raft_consensus.cc:2454] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1f63b79076f480eb77a58c04b2ed08e in term 1.
I20260808 06:57:30.984202 660 leader_election.cc:302] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9de4d11a44674988b2f2d5ef223b1b6a, a1f63b79076f480eb77a58c04b2ed08e; no voters:
I20260808 06:57:30.984279 882 raft_consensus.cc:2781] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:30.984459 882 raft_consensus.cc:686] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 1 LEADER]: Becoming Leader. State: Replica: a1f63b79076f480eb77a58c04b2ed08e, State: Running, Role: LEADER
I20260808 06:57:30.984555 882 consensus_queue.cc:237] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } }
I20260808 06:57:30.985445 32542 catalog_manager.cc:5495] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e reported cstate change: term changed from 0 to 1, leader changed from <none> to a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67). New cstate: current_term: 1 leader_uuid: "a1f63b79076f480eb77a58c04b2ed08e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } health_report { overall_health: UNKNOWN } } }
W20260808 06:57:31.006836 829 tablet.cc:2369] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:31.027029 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.024777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.21.28.67:41887)
0808 06:57:31.024845 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.024846 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.024853 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:31.024885 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.024886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.024903 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.024962 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:31.025108 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.025578 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.026302 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.026496 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.026551 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.026709 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.026810 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.026884 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.026929 (+ 45us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.026951 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 06:57:31.028875 551 raft_consensus.cc:1261] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Refusing update from remote peer a1f63b79076f480eb77a58c04b2ed08e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.028920 314 raft_consensus.cc:1261] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Refusing update from remote peer a1f63b79076f480eb77a58c04b2ed08e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.029188 882 consensus_queue.cc:1035] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.029332 886 consensus_queue.cc:1035] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.032501 895 mvcc.cc:205] Tried to move back new op lower bound from 7316161540211740672 to 7316161540033466368. Current Snapshot: MvccSnapshot[applied={T|T < 7316161540211740672}]
I20260808 06:57:31.032721 893 mvcc.cc:205] Tried to move back new op lower bound from 7316161540211740672 to 7316161540033466368. Current Snapshot: MvccSnapshot[applied={T|T < 7316161540211740672 or (T in {7316161540211740672})}]
I20260808 06:57:31.033744 32542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:31.033838 32542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:31.034535 32542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: Sending DeleteTablet for 3 replicas of tablet 3b7a1595bcbb41989e2ed0e8af74589c
I20260808 06:57:31.034827 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7a1595bcbb41989e2ed0e8af74589c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:39918
I20260808 06:57:31.034992 521 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7a1595bcbb41989e2ed0e8af74589c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:38500
I20260808 06:57:31.035212 910 tablet_replica.cc:330] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: stopping tablet replica
I20260808 06:57:31.035254 908 tablet_replica.cc:330] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: stopping tablet replica
I20260808 06:57:31.035315 32751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7a1595bcbb41989e2ed0e8af74589c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:60000
I20260808 06:57:31.035360 908 raft_consensus.cc:2229] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:31.035461 908 raft_consensus.cc:2258] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.035495 32542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:31.035617 32542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:31.035446 910 raft_consensus.cc:2229] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:31.035953 910 raft_consensus.cc:2258] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.036115 908 ts_tablet_manager.cc:1905] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.036514 910 ts_tablet_manager.cc:1905] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.037091 738 tablet_service.cc:1467] Processing CreateTablet for tablet 0079339d1e07445e83a9311f6f3a02f4 (DEFAULT_TABLE table=test-table [id=5c626d443f8d4e029566d10b3bd0bedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.037220 738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0079339d1e07445e83a9311f6f3a02f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.038023 32751 tablet_service.cc:1467] Processing CreateTablet for tablet 0079339d1e07445e83a9311f6f3a02f4 (DEFAULT_TABLE table=test-table [id=5c626d443f8d4e029566d10b3bd0bedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.038138 32751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0079339d1e07445e83a9311f6f3a02f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.038439 908 ts_tablet_manager.cc:1918] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:31.038509 908 log.cc:1196] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/3b7a1595bcbb41989e2ed0e8af74589c
I20260808 06:57:31.038582 910 ts_tablet_manager.cc:1918] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:31.038705 910 log.cc:1196] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/3b7a1595bcbb41989e2ed0e8af74589c
I20260808 06:57:31.038762 908 ts_tablet_manager.cc:1939] T 3b7a1595bcbb41989e2ed0e8af74589c P a1f63b79076f480eb77a58c04b2ed08e: Deleting consensus metadata
I20260808 06:57:31.039083 910 ts_tablet_manager.cc:1939] T 3b7a1595bcbb41989e2ed0e8af74589c P 7e086e9a1d1d4791842bc5464e70e61b: Deleting consensus metadata
I20260808 06:57:31.039103 912 tablet_replica.cc:330] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: stopping tablet replica
I20260808 06:57:31.039192 912 raft_consensus.cc:2229] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:31.039326 912 raft_consensus.cc:2258] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.039520 868 tablet_bootstrap.cc:492] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: Bootstrap starting.
I20260808 06:57:31.039819 912 ts_tablet_manager.cc:1905] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.039966 868 tablet_bootstrap.cc:654] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.040315 872 tablet_bootstrap.cc:492] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: Bootstrap starting.
I20260808 06:57:31.040689 872 tablet_bootstrap.cc:654] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.041134 32525 catalog_manager.cc:4841] TS a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887): tablet 3b7a1595bcbb41989e2ed0e8af74589c (table test-table [id=4d438e982c6b4b7f95d4860be052a845]) successfully deleted
I20260808 06:57:31.041219 912 ts_tablet_manager.cc:1918] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:31.041285 912 log.cc:1196] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/3b7a1595bcbb41989e2ed0e8af74589c
I20260808 06:57:31.041368 32522 catalog_manager.cc:4841] TS 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341): tablet 3b7a1595bcbb41989e2ed0e8af74589c (table test-table [id=4d438e982c6b4b7f95d4860be052a845]) successfully deleted
I20260808 06:57:31.041545 912 ts_tablet_manager.cc:1939] T 3b7a1595bcbb41989e2ed0e8af74589c P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting consensus metadata
I20260808 06:57:31.041590 521 tablet_service.cc:1467] Processing CreateTablet for tablet 0079339d1e07445e83a9311f6f3a02f4 (DEFAULT_TABLE table=test-table [id=5c626d443f8d4e029566d10b3bd0bedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.041692 521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0079339d1e07445e83a9311f6f3a02f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.042601 866 tablet_bootstrap.cc:492] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: Bootstrap starting.
I20260808 06:57:31.042660 32523 catalog_manager.cc:4841] TS 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003): tablet 3b7a1595bcbb41989e2ed0e8af74589c (table test-table [id=4d438e982c6b4b7f95d4860be052a845]) successfully deleted
I20260808 06:57:31.042845 872 tablet_bootstrap.cc:492] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: No bootstrap required, opened a new log
I20260808 06:57:31.042874 872 ts_tablet_manager.cc:1397] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260808 06:57:31.042987 872 raft_consensus.cc:348] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.043031 872 raft_consensus.cc:374] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.043021 866 tablet_bootstrap.cc:654] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.043044 872 raft_consensus.cc:729] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de4d11a44674988b2f2d5ef223b1b6a, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.043082 872 consensus_queue.cc:260] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.043377 872 ts_tablet_manager.cc:1428] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.043713 868 tablet_bootstrap.cc:492] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: No bootstrap required, opened a new log
I20260808 06:57:31.043763 868 ts_tablet_manager.cc:1397] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260808 06:57:31.043903 868 raft_consensus.cc:348] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.043990 868 raft_consensus.cc:374] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.044009 868 raft_consensus.cc:729] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f63b79076f480eb77a58c04b2ed08e, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.044046 868 consensus_queue.cc:260] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.044130 868 ts_tablet_manager.cc:1428] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.044549 866 tablet_bootstrap.cc:492] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: No bootstrap required, opened a new log
I20260808 06:57:31.044596 866 ts_tablet_manager.cc:1397] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:31.044745 866 raft_consensus.cc:348] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.044816 866 raft_consensus.cc:374] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.044834 866 raft_consensus.cc:729] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e086e9a1d1d4791842bc5464e70e61b, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.044883 866 consensus_queue.cc:260] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.044965 866 ts_tablet_manager.cc:1428] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:31.061156 881 raft_consensus.cc:482] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:31.061259 881 raft_consensus.cc:504] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.061600 881 leader_election.cc:288] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003), a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887)
I20260808 06:57:31.065033 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.061757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:41887 (local address 127.21.28.66:43451)
0808 06:57:31.061816 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.061828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.061869 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.062116 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.062118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.062120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:31.062207 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.062210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.062948 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.062949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.064062 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.064064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.064797 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.064798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.064871 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.064876 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.064930 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.064976 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 06:57:31.065259 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.061946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:43451 (local address 127.21.28.67:41887)
0808 06:57:31.061994 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.061996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.062003 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:31.062028 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.062028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.062038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.062084 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:31.062300 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.062878 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.064154 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.064712 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.064802 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.064813 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.065127 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.065193 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 06:57:31.065307 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.061765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40003 (local address 127.21.28.66:49109)
0808 06:57:31.061938 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.061947 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.061987 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.062331 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.062331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.062333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:31.062418 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.062420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.063346 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.063347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.064375 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.064377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.064968 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.064969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.065047 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.065052 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.065157 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.065209 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":21,"thread_start_us":109,"threads_started":1}
I20260808 06:57:31.065511 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0079339d1e07445e83a9311f6f3a02f4" candidate_uuid: "7e086e9a1d1d4791842bc5464e70e61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f63b79076f480eb77a58c04b2ed08e" is_pre_election: true
I20260808 06:57:31.065646 765 raft_consensus.cc:2454] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e086e9a1d1d4791842bc5464e70e61b in term 0.
I20260808 06:57:31.065729 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.062140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:49109 (local address 127.21.28.65:40003)
0808 06:57:31.062192 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.062194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.062201 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:31.062224 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.062224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.062241 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.062285 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:31.062502 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.063234 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.064444 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.064854 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.064930 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.064940 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.065615 (+ 675us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.065662 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 06:57:31.065888 440 leader_election.cc:302] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e086e9a1d1d4791842bc5464e70e61b, a1f63b79076f480eb77a58c04b2ed08e; no voters:
I20260808 06:57:31.065905 314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0079339d1e07445e83a9311f6f3a02f4" candidate_uuid: "7e086e9a1d1d4791842bc5464e70e61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" is_pre_election: true
I20260808 06:57:31.065979 314 raft_consensus.cc:2454] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e086e9a1d1d4791842bc5464e70e61b in term 0.
I20260808 06:57:31.066061 881 raft_consensus.cc:2781] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:31.066116 881 raft_consensus.cc:482] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:31.066151 881 raft_consensus.cc:3037] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.066962 881 raft_consensus.cc:504] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.067072 881 leader_election.cc:288] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [CANDIDATE]: Term 1 election: Requested vote from peers 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003), a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887)
I20260808 06:57:31.067237 314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0079339d1e07445e83a9311f6f3a02f4" candidate_uuid: "7e086e9a1d1d4791842bc5464e70e61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de4d11a44674988b2f2d5ef223b1b6a"
I20260808 06:57:31.067281 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0079339d1e07445e83a9311f6f3a02f4" candidate_uuid: "7e086e9a1d1d4791842bc5464e70e61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f63b79076f480eb77a58c04b2ed08e"
I20260808 06:57:31.067322 314 raft_consensus.cc:3037] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.067340 765 raft_consensus.cc:3037] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.068125 765 raft_consensus.cc:2454] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e086e9a1d1d4791842bc5464e70e61b in term 1.
I20260808 06:57:31.068125 314 raft_consensus.cc:2454] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e086e9a1d1d4791842bc5464e70e61b in term 1.
I20260808 06:57:31.068301 437 leader_election.cc:302] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e086e9a1d1d4791842bc5464e70e61b, 9de4d11a44674988b2f2d5ef223b1b6a; no voters:
I20260808 06:57:31.068385 881 raft_consensus.cc:2781] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:31.068521 881 raft_consensus.cc:686] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 LEADER]: Becoming Leader. State: Replica: 7e086e9a1d1d4791842bc5464e70e61b, State: Running, Role: LEADER
I20260808 06:57:31.068593 881 consensus_queue.cc:237] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.069239 32542 catalog_manager.cc:5495] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66). New cstate: current_term: 1 leader_uuid: "7e086e9a1d1d4791842bc5464e70e61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } health_report { overall_health: UNKNOWN } } }
Aug 08 06:57:31 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172250, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.126@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:31.102790 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.092413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:42223 (local address 127.0.0.1:60916)
0808 06:57:31.092588 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.092600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.092675 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.092829 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.092830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.093244 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:31.093305 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.093308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.093985 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.093986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.094804 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.094807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.095165 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.095166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.095222 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.098409 (+ 3187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:31.098415 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:31.100071 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:31.101704 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:31.101712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:31.101714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:31.101951 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:31.102157 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:31.102158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:31.102159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:31.102225 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:31.102483 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:31.102489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:31.102530 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.102577 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.102672 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260808 06:57:31.102860 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.092500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.21.28.126:42223)
0808 06:57:31.092555 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.092558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.092671 (+ 113us) server_negotiation.cc:407] Connection header received
0808 06:57:31.092743 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.092744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.092754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.092812 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:31.093388 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.093909 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.094898 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.095089 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.095150 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.100155 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:31.100157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:31.100157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:31.100164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:31.101612 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:31.102023 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:31.102024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:31.102025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:31.102087 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:31.102286 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:31.102287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:31.102287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:31.102422 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:31.102485 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.102618 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.102739 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":32}
I20260808 06:57:31.104948 32542 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:60916:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 06:57:31.105170 32542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:31.106634 521 tablet_service.cc:1467] Processing CreateTablet for tablet ff4daa5ee3844c3dac0e83371e1378dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23ca89f8f0074da4b2035707558278c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:31.106665 738 tablet_service.cc:1467] Processing CreateTablet for tablet ff4daa5ee3844c3dac0e83371e1378dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23ca89f8f0074da4b2035707558278c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:31.106665 32751 tablet_service.cc:1467] Processing CreateTablet for tablet ff4daa5ee3844c3dac0e83371e1378dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23ca89f8f0074da4b2035707558278c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:57:31.106803 738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4daa5ee3844c3dac0e83371e1378dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.106804 521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4daa5ee3844c3dac0e83371e1378dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.106824 32751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4daa5ee3844c3dac0e83371e1378dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.108386 868 tablet_bootstrap.cc:492] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: Bootstrap starting.
I20260808 06:57:31.108546 872 tablet_bootstrap.cc:492] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: Bootstrap starting.
I20260808 06:57:31.108771 868 tablet_bootstrap.cc:654] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.109333 866 tablet_bootstrap.cc:492] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: Bootstrap starting.
I20260808 06:57:31.109699 866 tablet_bootstrap.cc:654] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.110186 872 tablet_bootstrap.cc:654] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.110277 866 tablet_bootstrap.cc:492] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: No bootstrap required, opened a new log
I20260808 06:57:31.110334 866 ts_tablet_manager.cc:1397] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:31.110481 866 raft_consensus.cc:348] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.110539 866 raft_consensus.cc:374] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.110566 866 raft_consensus.cc:729] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e086e9a1d1d4791842bc5464e70e61b, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.110615 866 consensus_queue.cc:260] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.110714 866 ts_tablet_manager.cc:1428] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:31.110836 881 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.110905 881 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:31.111001 872 tablet_bootstrap.cc:492] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: No bootstrap required, opened a new log
I20260808 06:57:31.111055 872 ts_tablet_manager.cc:1397] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260808 06:57:31.111196 868 tablet_bootstrap.cc:492] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: No bootstrap required, opened a new log
I20260808 06:57:31.111200 872 raft_consensus.cc:348] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.111259 868 ts_tablet_manager.cc:1397] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 06:57:31.111291 872 raft_consensus.cc:374] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.111320 872 raft_consensus.cc:729] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de4d11a44674988b2f2d5ef223b1b6a, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.111361 872 consensus_queue.cc:260] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.111407 868 raft_consensus.cc:348] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.111464 872 ts_tablet_manager.cc:1428] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.111488 868 raft_consensus.cc:374] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.111514 868 raft_consensus.cc:729] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f63b79076f480eb77a58c04b2ed08e, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.111554 868 consensus_queue.cc:260] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.111637 868 ts_tablet_manager.cc:1428] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.111770 904 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.111843 904 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:31.111618 883 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.112092 883 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:31.122843 883 raft_consensus.cc:482] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:31.122927 883 raft_consensus.cc:504] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.123224 883 leader_election.cc:288] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341), a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887)
I20260808 06:57:31.126396 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.123402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46341 (local address 127.21.28.65:42427)
0808 06:57:31.123664 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.123699 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.123737 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.123976 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.123976 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.123999 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:31.124050 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.124051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.124721 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.124723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.125628 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.125633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.126140 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.126141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.126220 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.126224 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.126274 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.126313 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":117,"threads_started":1}
I20260808 06:57:31.126405 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.123778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:42427 (local address 127.21.28.66:46341)
0808 06:57:31.123841 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.123843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.123849 (+ 6us) server_negotiation.cc:407] Connection header received
0808 06:57:31.123869 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.123870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.123876 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.123925 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:31.124135 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.124649 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.125713 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.126073 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.126132 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.126142 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.126286 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.126337 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:57:31.126907 551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4daa5ee3844c3dac0e83371e1378dd" candidate_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e086e9a1d1d4791842bc5464e70e61b" is_pre_election: true
I20260808 06:57:31.127014 551 raft_consensus.cc:2454] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9de4d11a44674988b2f2d5ef223b1b6a in term 0.
I20260808 06:57:31.127059 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.123630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:35717 (local address 127.21.28.67:41887)
0808 06:57:31.123816 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.123818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.123828 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:31.123852 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.123853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.123863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.123925 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:31.124099 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.124801 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.126215 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.126732 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.126818 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.126827 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.126950 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.126995 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:31.127173 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.123395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:41887 (local address 127.21.28.65:35717)
0808 06:57:31.123589 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.123600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.123655 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.123938 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.123939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.123941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:31.124027 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.124029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.124960 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.124961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.126095 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.126099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.126817 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.126818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.126889 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.126892 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.126943 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.126978 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":16,"thread_start_us":128,"threads_started":1}
I20260808 06:57:31.127386 32679 leader_election.cc:302] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e086e9a1d1d4791842bc5464e70e61b, 9de4d11a44674988b2f2d5ef223b1b6a; no voters:
I20260808 06:57:31.127424 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4daa5ee3844c3dac0e83371e1378dd" candidate_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f63b79076f480eb77a58c04b2ed08e" is_pre_election: true
I20260808 06:57:31.127504 765 raft_consensus.cc:2454] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9de4d11a44674988b2f2d5ef223b1b6a in term 0.
I20260808 06:57:31.127514 883 raft_consensus.cc:2781] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:31.127543 883 raft_consensus.cc:482] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:31.127562 883 raft_consensus.cc:3037] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.128517 883 raft_consensus.cc:504] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.128659 883 leader_election.cc:288] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [CANDIDATE]: Term 1 election: Requested vote from peers 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341), a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887)
I20260808 06:57:31.128803 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4daa5ee3844c3dac0e83371e1378dd" candidate_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f63b79076f480eb77a58c04b2ed08e"
I20260808 06:57:31.128890 765 raft_consensus.cc:3037] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.128890 551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4daa5ee3844c3dac0e83371e1378dd" candidate_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e086e9a1d1d4791842bc5464e70e61b"
I20260808 06:57:31.128974 551 raft_consensus.cc:3037] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.129552 765 raft_consensus.cc:2454] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9de4d11a44674988b2f2d5ef223b1b6a in term 1.
I20260808 06:57:31.129792 551 raft_consensus.cc:2454] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9de4d11a44674988b2f2d5ef223b1b6a in term 1.
I20260808 06:57:31.129876 32684 leader_election.cc:302] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9de4d11a44674988b2f2d5ef223b1b6a, a1f63b79076f480eb77a58c04b2ed08e; no voters:
I20260808 06:57:31.129992 883 raft_consensus.cc:2781] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:31.130162 883 raft_consensus.cc:686] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 LEADER]: Becoming Leader. State: Replica: 9de4d11a44674988b2f2d5ef223b1b6a, State: Running, Role: LEADER
I20260808 06:57:31.130260 883 consensus_queue.cc:237] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.130606 939 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: TxnStatusTablet state changed. Reason: New leader 9de4d11a44674988b2f2d5ef223b1b6a. Latest consensus state: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.130764 939 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:31.130993 942 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 06:57:31.130944 32542 catalog_manager.cc:5495] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65). New cstate: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } health_report { overall_health: UNKNOWN } } }
I20260808 06:57:31.131100 942 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 06:57:31.139684 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.137014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40003 (local address 127.0.0.1:60010)
0808 06:57:31.137133 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.137144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.137197 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.137341 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.137342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.137343 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:31.137398 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.137401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.138000 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.138001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.138722 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.138725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.139119 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.139120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.139192 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.139207 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.139459 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.139468 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.139510 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.139564 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 06:57:31.139829 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.137069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.21.28.65:40003)
0808 06:57:31.137114 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.137116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.137178 (+ 62us) server_negotiation.cc:407] Connection header received
0808 06:57:31.137265 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.137266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.137286 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.137340 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:31.137445 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.137921 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.138795 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.139049 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.139103 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.139299 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.139395 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.139468 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.139740 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.139767 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:57:31.140879 551 raft_consensus.cc:1261] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Refusing update from remote peer 9de4d11a44674988b2f2d5ef223b1b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.141136 765 raft_consensus.cc:1261] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Refusing update from remote peer 9de4d11a44674988b2f2d5ef223b1b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.141418 883 consensus_queue.cc:1035] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.141597 939 consensus_queue.cc:1035] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.142028 881 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: TxnStatusTablet state changed. Reason: New leader 9de4d11a44674988b2f2d5ef223b1b6a. Latest consensus state: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.142118 881 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:31.142202 904 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: TxnStatusTablet state changed. Reason: New leader 9de4d11a44674988b2f2d5ef223b1b6a. Latest consensus state: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.142277 904 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:31.142935 939 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.143010 939 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:31.143376 904 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.143429 892 mvcc.cc:205] Tried to move back new op lower bound from 7316161540671180800 to 7316161540630122496. Current Snapshot: MvccSnapshot[applied={T|T < 7316161540671180800}]
I20260808 06:57:31.143463 904 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P a1f63b79076f480eb77a58c04b2ed08e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:31.144572 883 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.144652 883 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P 9de4d11a44674988b2f2d5ef223b1b6a: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:57:31.145403 881 tablet_replica.cc:439] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } } }
I20260808 06:57:31.145485 881 tablet_replica.cc:442] T ff4daa5ee3844c3dac0e83371e1378dd P 7e086e9a1d1d4791842bc5464e70e61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:57:31.147434 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.144949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.21.28.66:46341)
0808 06:57:31.145002 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.145003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.145010 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:31.145046 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.145047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.145070 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.145128 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:31.145300 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.145938 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.146703 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.146899 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.146953 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.147093 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.147233 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.147294 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.147338 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.147359 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:57:31.151762 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.149672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:58465 (local address 127.21.28.65:40003)
0808 06:57:31.149711 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.149713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.149725 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:57:31.149786 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.149787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.149796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.149846 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:31.149961 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.150408 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.151104 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.151318 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.151372 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.151525 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.151557 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.151600 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.151649 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.151696 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:31.152061 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.149523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40003 (local address 127.21.28.66:58465)
0808 06:57:31.149670 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.149681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.149738 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.149849 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.149850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.149851 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:31.149904 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.149906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.150474 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.150475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.151035 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.151037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.151379 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.151380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.151446 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.151460 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.151606 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.151607 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.151648 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.151709 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260808 06:57:31.154203 765 raft_consensus.cc:1261] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Refusing update from remote peer 7e086e9a1d1d4791842bc5464e70e61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.154333 314 raft_consensus.cc:1261] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Refusing update from remote peer 7e086e9a1d1d4791842bc5464e70e61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.154582 917 consensus_queue.cc:1035] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.154744 881 consensus_queue.cc:1035] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.175521 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.172081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:46341 (local address 127.21.28.65:43051)
0808 06:57:31.172273 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.172282 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.172341 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.172462 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.172463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.172464 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:31.172559 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.172561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.173239 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.173240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.174680 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.174684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.175085 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.175086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.175155 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.175174 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.175333 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.175335 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.175372 (+ 37us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.175416 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
I20260808 06:57:31.175503 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.172146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:43051 (local address 127.21.28.66:46341)
0808 06:57:31.172194 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.172196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.172348 (+ 152us) server_negotiation.cc:407] Connection header received
0808 06:57:31.172394 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.172395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.172406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.172456 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:31.172614 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.173168 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.174758 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.175022 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.175079 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.175247 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.175285 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.175329 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.175384 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.175416 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:31.186075 32541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:31.186199 32541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:31.186815 32541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: Sending DeleteTablet for 3 replicas of tablet 0079339d1e07445e83a9311f6f3a02f4
I20260808 06:57:31.187100 32751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0079339d1e07445e83a9311f6f3a02f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:60000
I20260808 06:57:31.187181 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 0079339d1e07445e83a9311f6f3a02f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:39918
I20260808 06:57:31.187211 912 tablet_replica.cc:330] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: stopping tablet replica
I20260808 06:57:31.187269 912 raft_consensus.cc:2229] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:31.187275 908 tablet_replica.cc:330] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: stopping tablet replica
I20260808 06:57:31.187319 912 raft_consensus.cc:2258] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.187328 908 raft_consensus.cc:2229] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:31.187361 908 raft_consensus.cc:2258] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.187489 521 tablet_service.cc:1514] Processing DeleteTablet for tablet 0079339d1e07445e83a9311f6f3a02f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:38500
I20260808 06:57:31.187583 910 tablet_replica.cc:330] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: stopping tablet replica
I20260808 06:57:31.187638 910 raft_consensus.cc:2229] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:31.187767 910 raft_consensus.cc:2258] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.187819 912 ts_tablet_manager.cc:1905] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.188077 910 ts_tablet_manager.cc:1905] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.189576 912 ts_tablet_manager.cc:1918] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:31.189627 912 log.cc:1196] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/0079339d1e07445e83a9311f6f3a02f4
I20260808 06:57:31.189713 910 ts_tablet_manager.cc:1918] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:31.189764 910 log.cc:1196] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/0079339d1e07445e83a9311f6f3a02f4
I20260808 06:57:31.189833 912 ts_tablet_manager.cc:1939] T 0079339d1e07445e83a9311f6f3a02f4 P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting consensus metadata
I20260808 06:57:31.189968 910 ts_tablet_manager.cc:1939] T 0079339d1e07445e83a9311f6f3a02f4 P 7e086e9a1d1d4791842bc5464e70e61b: Deleting consensus metadata
I20260808 06:57:31.190198 908 ts_tablet_manager.cc:1905] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.191803 908 ts_tablet_manager.cc:1918] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:31.191864 908 log.cc:1196] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/0079339d1e07445e83a9311f6f3a02f4
I20260808 06:57:31.192075 908 ts_tablet_manager.cc:1939] T 0079339d1e07445e83a9311f6f3a02f4 P a1f63b79076f480eb77a58c04b2ed08e: Deleting consensus metadata
I20260808 06:57:31.192404 32522 catalog_manager.cc:4841] TS 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341): tablet 0079339d1e07445e83a9311f6f3a02f4 (table test-table [id=5c626d443f8d4e029566d10b3bd0bedf]) successfully deleted
I20260808 06:57:31.192404 32525 catalog_manager.cc:4841] TS a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887): tablet 0079339d1e07445e83a9311f6f3a02f4 (table test-table [id=5c626d443f8d4e029566d10b3bd0bedf]) successfully deleted
I20260808 06:57:31.195441 32523 catalog_manager.cc:4841] TS 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003): tablet 0079339d1e07445e83a9311f6f3a02f4 (table test-table [id=5c626d443f8d4e029566d10b3bd0bedf]) successfully deleted
Aug 08 06:57:31 dist-test-slave-nj21 krb5kdc[32484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 06:57:31 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:31 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:57:31.221925 32614 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:31.215074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.21.28.126:42223)
0808 06:57:31.215121 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.215123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.215288 (+ 165us) server_negotiation.cc:407] Connection header received
0808 06:57:31.215349 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.215351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.215363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.215424 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:31.216162 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.216952 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.218082 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.218373 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.218440 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.221851 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60922: BlockingRecv error: recv got EOF from 127.0.0.1:60922 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 08 06:57:31 dist-test-slave-nj21 krb5kdc[32484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM
I20260808 06:57:31.231447 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.223202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.21.28.126:42223)
0808 06:57:31.223243 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.223245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.223299 (+ 54us) server_negotiation.cc:407] Connection header received
0808 06:57:31.223351 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.223352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.223361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.223421 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:31.224037 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.224766 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.225719 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.226024 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.226089 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.228669 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:31.228671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:31.228671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:31.228678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:31.230290 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:31.230664 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:31.230665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:31.230665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:31.230730 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:31.230918 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:31.230919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:31.230919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:31.231057 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:31.231157 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.231270 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.231348 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:31.233299 32541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:31.233471 32541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:31.235024 32751 tablet_service.cc:1467] Processing CreateTablet for tablet 27f3b9caaed54f30bbf332e14716888a (DEFAULT_TABLE table=test-table [id=a1e591ac1705440bbb7ff55ba1a7b53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.235025 521 tablet_service.cc:1467] Processing CreateTablet for tablet 27f3b9caaed54f30bbf332e14716888a (DEFAULT_TABLE table=test-table [id=a1e591ac1705440bbb7ff55ba1a7b53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.235167 32751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f3b9caaed54f30bbf332e14716888a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.235167 521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f3b9caaed54f30bbf332e14716888a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.236711 866 tablet_bootstrap.cc:492] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: Bootstrap starting.
I20260808 06:57:31.236718 738 tablet_service.cc:1467] Processing CreateTablet for tablet 27f3b9caaed54f30bbf332e14716888a (DEFAULT_TABLE table=test-table [id=a1e591ac1705440bbb7ff55ba1a7b53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.236843 738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f3b9caaed54f30bbf332e14716888a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.237133 866 tablet_bootstrap.cc:654] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.238114 868 tablet_bootstrap.cc:492] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: Bootstrap starting.
I20260808 06:57:31.238185 872 tablet_bootstrap.cc:492] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Bootstrap starting.
I20260808 06:57:31.238471 868 tablet_bootstrap.cc:654] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.238631 872 tablet_bootstrap.cc:654] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.239287 868 tablet_bootstrap.cc:492] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: No bootstrap required, opened a new log
I20260808 06:57:31.239336 868 ts_tablet_manager.cc:1397] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:31.239442 868 raft_consensus.cc:348] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.239504 868 raft_consensus.cc:374] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.239524 868 raft_consensus.cc:729] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f63b79076f480eb77a58c04b2ed08e, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.239571 868 consensus_queue.cc:260] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.239660 868 ts_tablet_manager.cc:1428] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.240367 872 tablet_bootstrap.cc:492] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: No bootstrap required, opened a new log
I20260808 06:57:31.240417 872 ts_tablet_manager.cc:1397] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260808 06:57:31.240558 872 raft_consensus.cc:348] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.240618 872 raft_consensus.cc:374] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.240635 872 raft_consensus.cc:729] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de4d11a44674988b2f2d5ef223b1b6a, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.240671 872 consensus_queue.cc:260] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.240757 872 ts_tablet_manager.cc:1428] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.241840 866 tablet_bootstrap.cc:492] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: No bootstrap required, opened a new log
I20260808 06:57:31.241894 866 ts_tablet_manager.cc:1397] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 06:57:31.242178 866 raft_consensus.cc:348] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.242278 866 raft_consensus.cc:374] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.242300 866 raft_consensus.cc:729] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e086e9a1d1d4791842bc5464e70e61b, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.242340 866 consensus_queue.cc:260] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.242434 866 ts_tablet_manager.cc:1428] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:31.247773 904 raft_consensus.cc:482] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:31.247857 904 raft_consensus.cc:504] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.248018 904 leader_election.cc:288] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003), 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341)
I20260808 06:57:31.248174 551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f3b9caaed54f30bbf332e14716888a" candidate_uuid: "a1f63b79076f480eb77a58c04b2ed08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e086e9a1d1d4791842bc5464e70e61b" is_pre_election: true
I20260808 06:57:31.248219 314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f3b9caaed54f30bbf332e14716888a" candidate_uuid: "a1f63b79076f480eb77a58c04b2ed08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" is_pre_election: true
I20260808 06:57:31.248274 551 raft_consensus.cc:2454] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1f63b79076f480eb77a58c04b2ed08e in term 0.
I20260808 06:57:31.248298 314 raft_consensus.cc:2454] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1f63b79076f480eb77a58c04b2ed08e in term 0.
I20260808 06:57:31.248422 660 leader_election.cc:302] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9de4d11a44674988b2f2d5ef223b1b6a, a1f63b79076f480eb77a58c04b2ed08e; no voters:
I20260808 06:57:31.248541 904 raft_consensus.cc:2781] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:31.248572 904 raft_consensus.cc:482] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:31.248594 904 raft_consensus.cc:3037] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.249461 904 raft_consensus.cc:504] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.249593 904 leader_election.cc:288] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [CANDIDATE]: Term 1 election: Requested vote from peers 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003), 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341)
I20260808 06:57:31.249756 551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f3b9caaed54f30bbf332e14716888a" candidate_uuid: "a1f63b79076f480eb77a58c04b2ed08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e086e9a1d1d4791842bc5464e70e61b"
I20260808 06:57:31.249779 314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f3b9caaed54f30bbf332e14716888a" candidate_uuid: "a1f63b79076f480eb77a58c04b2ed08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de4d11a44674988b2f2d5ef223b1b6a"
I20260808 06:57:31.249823 551 raft_consensus.cc:3037] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.249836 314 raft_consensus.cc:3037] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.250495 314 raft_consensus.cc:2454] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1f63b79076f480eb77a58c04b2ed08e in term 1.
I20260808 06:57:31.250595 551 raft_consensus.cc:2454] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1f63b79076f480eb77a58c04b2ed08e in term 1.
I20260808 06:57:31.250640 660 leader_election.cc:302] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9de4d11a44674988b2f2d5ef223b1b6a, a1f63b79076f480eb77a58c04b2ed08e; no voters:
I20260808 06:57:31.250706 904 raft_consensus.cc:2781] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:31.250738 904 raft_consensus.cc:686] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 1 LEADER]: Becoming Leader. State: Replica: a1f63b79076f480eb77a58c04b2ed08e, State: Running, Role: LEADER
I20260808 06:57:31.250777 904 consensus_queue.cc:237] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.251340 32542 catalog_manager.cc:5495] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e reported cstate change: term changed from 0 to 1, leader changed from <none> to a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67). New cstate: current_term: 1 leader_uuid: "a1f63b79076f480eb77a58c04b2ed08e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } health_report { overall_health: HEALTHY } } }
I20260808 06:57:31.260531 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.257595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.21.28.67:41887)
0808 06:57:31.257635 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.257637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.257651 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:57:31.257674 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.257675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.257685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.257738 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:31.257927 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.258647 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.259754 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.260052 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.260125 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.260291 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.260327 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.260381 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.260434 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.260457 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:57:31.261193 551 raft_consensus.cc:1261] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Refusing update from remote peer a1f63b79076f480eb77a58c04b2ed08e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.261252 314 raft_consensus.cc:1261] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Refusing update from remote peer a1f63b79076f480eb77a58c04b2ed08e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.261442 904 consensus_queue.cc:1035] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.261562 904 consensus_queue.cc:1035] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.264758 32542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:31.264855 32542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:31.265470 32542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: Sending DeleteTablet for 3 replicas of tablet 27f3b9caaed54f30bbf332e14716888a
I20260808 06:57:31.265756 521 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f3b9caaed54f30bbf332e14716888a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:38500
I20260808 06:57:31.265844 910 tablet_replica.cc:330] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: stopping tablet replica
I20260808 06:57:31.265893 910 raft_consensus.cc:2229] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:31.265951 910 raft_consensus.cc:2258] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.266232 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f3b9caaed54f30bbf332e14716888a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:39918
I20260808 06:57:31.266244 32751 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f3b9caaed54f30bbf332e14716888a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:60000
I20260808 06:57:31.266320 908 tablet_replica.cc:330] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: stopping tablet replica
I20260808 06:57:31.266376 908 raft_consensus.cc:2229] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:31.266441 908 raft_consensus.cc:2258] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.266525 912 tablet_replica.cc:330] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: stopping tablet replica
I20260808 06:57:31.266592 912 raft_consensus.cc:2229] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:31.266633 912 pending_rounds.cc:62] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Trying to abort 2 pending ops.
I20260808 06:57:31.266657 912 pending_rounds.cc:68] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316161541164249088 op_type: WRITE_OP write_request { tablet_id: "27f3b9caaed54f30bbf332e14716888a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\207\251\333\323\006\"8\n\ntest-admin\022*\n a1e591ac1705440bbb7ff55ba1a7b53e\020\001\030\001 \001(\001" signature: "<redacted>""%>\257\3219\221&\013\213\033B\334S\346\246\233878e\016RJ\355\253\371\276\341$W\203:\337\366<\n\342\242T\365T\037\204j\211\r\341\025\233K\274\251\222\252g\250t>\r00\021\347\360" signing_key_seq_num: 0 } } request_id { client_id: "6777ceac41bb49ed852df3690c79fb57" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260808 06:57:31.266839 912 pending_rounds.cc:68] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316161541123371008 op_type: NO_OP noop_request { }
I20260808 06:57:31.266884 912 raft_consensus.cc:2866] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 06:57:31.266933 912 raft_consensus.cc:2258] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.267092 32542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:31.267201 984 consensus_queue.cc:1159] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
W20260808 06:57:31.267236 32542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:31.267665 910 ts_tablet_manager.cc:1905] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.269271 910 ts_tablet_manager.cc:1918] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:31.269335 910 log.cc:1196] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/27f3b9caaed54f30bbf332e14716888a
I20260808 06:57:31.269556 910 ts_tablet_manager.cc:1939] T 27f3b9caaed54f30bbf332e14716888a P 7e086e9a1d1d4791842bc5464e70e61b: Deleting consensus metadata
I20260808 06:57:31.269877 32522 catalog_manager.cc:4841] TS 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341): tablet 27f3b9caaed54f30bbf332e14716888a (table test-table [id=a1e591ac1705440bbb7ff55ba1a7b53e]) successfully deleted
I20260808 06:57:31.270089 521 tablet_service.cc:1467] Processing CreateTablet for tablet 9851565a3e7b4ac89dbc0da0d36402f9 (DEFAULT_TABLE table=test-table [id=74b1a12435dc4c0383c7b10d5bfc9ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.270197 521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9851565a3e7b4ac89dbc0da0d36402f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.271389 866 tablet_bootstrap.cc:492] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: Bootstrap starting.
I20260808 06:57:31.271616 32751 tablet_service.cc:1467] Processing CreateTablet for tablet 9851565a3e7b4ac89dbc0da0d36402f9 (DEFAULT_TABLE table=test-table [id=74b1a12435dc4c0383c7b10d5bfc9ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.271837 866 tablet_bootstrap.cc:654] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.271850 908 ts_tablet_manager.cc:1905] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.273432 908 ts_tablet_manager.cc:1918] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:31.273525 908 log.cc:1196] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/27f3b9caaed54f30bbf332e14716888a
I20260808 06:57:31.273743 908 ts_tablet_manager.cc:1939] T 27f3b9caaed54f30bbf332e14716888a P a1f63b79076f480eb77a58c04b2ed08e: Deleting consensus metadata
I20260808 06:57:31.274127 32525 catalog_manager.cc:4841] TS a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887): tablet 27f3b9caaed54f30bbf332e14716888a (table test-table [id=a1e591ac1705440bbb7ff55ba1a7b53e]) successfully deleted
I20260808 06:57:31.274195 738 tablet_service.cc:1467] Processing CreateTablet for tablet 9851565a3e7b4ac89dbc0da0d36402f9 (DEFAULT_TABLE table=test-table [id=74b1a12435dc4c0383c7b10d5bfc9ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:31.274303 738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9851565a3e7b4ac89dbc0da0d36402f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.275504 868 tablet_bootstrap.cc:492] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: Bootstrap starting.
I20260808 06:57:31.275894 868 tablet_bootstrap.cc:654] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.276039 912 ts_tablet_manager.cc:1905] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.276289 32751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9851565a3e7b4ac89dbc0da0d36402f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:31.277534 912 ts_tablet_manager.cc:1918] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:31.277585 912 log.cc:1196] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/27f3b9caaed54f30bbf332e14716888a
I20260808 06:57:31.277639 872 tablet_bootstrap.cc:492] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a: Bootstrap starting.
I20260808 06:57:31.277778 912 ts_tablet_manager.cc:1939] T 27f3b9caaed54f30bbf332e14716888a P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting consensus metadata
I20260808 06:57:31.278019 872 tablet_bootstrap.cc:654] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:31.278441 32523 catalog_manager.cc:4841] TS 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003): tablet 27f3b9caaed54f30bbf332e14716888a (table test-table [id=a1e591ac1705440bbb7ff55ba1a7b53e]) successfully deleted
I20260808 06:57:31.279036 868 tablet_bootstrap.cc:492] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: No bootstrap required, opened a new log
I20260808 06:57:31.279079 868 ts_tablet_manager.cc:1397] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260808 06:57:31.279182 868 raft_consensus.cc:348] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.279248 868 raft_consensus.cc:374] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.279268 868 raft_consensus.cc:729] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f63b79076f480eb77a58c04b2ed08e, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.279310 868 consensus_queue.cc:260] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.279394 868 ts_tablet_manager.cc:1428] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.281838 866 tablet_bootstrap.cc:492] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: No bootstrap required, opened a new log
I20260808 06:57:31.281884 866 ts_tablet_manager.cc:1397] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260808 06:57:31.282028 866 raft_consensus.cc:348] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.282086 866 raft_consensus.cc:374] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.282105 866 raft_consensus.cc:729] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e086e9a1d1d4791842bc5464e70e61b, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.282150 866 consensus_queue.cc:260] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.282238 866 ts_tablet_manager.cc:1428] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.284204 872 tablet_bootstrap.cc:492] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a: No bootstrap required, opened a new log
I20260808 06:57:31.284263 872 ts_tablet_manager.cc:1397] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.000s
I20260808 06:57:31.284380 872 raft_consensus.cc:348] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.284437 872 raft_consensus.cc:374] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:31.284453 872 raft_consensus.cc:729] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de4d11a44674988b2f2d5ef223b1b6a, State: Initialized, Role: FOLLOWER
I20260808 06:57:31.284489 872 consensus_queue.cc:260] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.284561 872 ts_tablet_manager.cc:1428] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:31.672665 939 raft_consensus.cc:482] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:31.672775 939 raft_consensus.cc:504] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.672916 939 leader_election.cc:288] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341), a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887)
I20260808 06:57:31.673174 551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9851565a3e7b4ac89dbc0da0d36402f9" candidate_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e086e9a1d1d4791842bc5464e70e61b" is_pre_election: true
I20260808 06:57:31.673177 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9851565a3e7b4ac89dbc0da0d36402f9" candidate_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f63b79076f480eb77a58c04b2ed08e" is_pre_election: true
I20260808 06:57:31.673282 765 raft_consensus.cc:2454] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9de4d11a44674988b2f2d5ef223b1b6a in term 0.
I20260808 06:57:31.673282 551 raft_consensus.cc:2454] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9de4d11a44674988b2f2d5ef223b1b6a in term 0.
I20260808 06:57:31.673450 32679 leader_election.cc:302] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e086e9a1d1d4791842bc5464e70e61b, 9de4d11a44674988b2f2d5ef223b1b6a; no voters:
I20260808 06:57:31.673583 939 raft_consensus.cc:2781] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:31.673635 939 raft_consensus.cc:482] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:31.673662 939 raft_consensus.cc:3037] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.674486 939 raft_consensus.cc:504] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.674610 939 leader_election.cc:288] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [CANDIDATE]: Term 1 election: Requested vote from peers 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341), a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887)
I20260808 06:57:31.674770 551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9851565a3e7b4ac89dbc0da0d36402f9" candidate_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e086e9a1d1d4791842bc5464e70e61b"
I20260808 06:57:31.674876 551 raft_consensus.cc:3037] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.674854 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9851565a3e7b4ac89dbc0da0d36402f9" candidate_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f63b79076f480eb77a58c04b2ed08e"
I20260808 06:57:31.674995 765 raft_consensus.cc:3037] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:31.675663 765 raft_consensus.cc:2454] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9de4d11a44674988b2f2d5ef223b1b6a in term 1.
I20260808 06:57:31.675850 551 raft_consensus.cc:2454] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9de4d11a44674988b2f2d5ef223b1b6a in term 1.
I20260808 06:57:31.675984 32684 leader_election.cc:302] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9de4d11a44674988b2f2d5ef223b1b6a, a1f63b79076f480eb77a58c04b2ed08e; no voters:
I20260808 06:57:31.676128 939 raft_consensus.cc:2781] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:31.676203 939 raft_consensus.cc:686] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 LEADER]: Becoming Leader. State: Replica: 9de4d11a44674988b2f2d5ef223b1b6a, State: Running, Role: LEADER
I20260808 06:57:31.676251 939 consensus_queue.cc:237] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } }
I20260808 06:57:31.676823 32542 catalog_manager.cc:5495] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65). New cstate: current_term: 1 leader_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9de4d11a44674988b2f2d5ef223b1b6a" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 40003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 } health_report { overall_health: UNKNOWN } } }
I20260808 06:57:31.717996 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.715857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.21.28.65:40003)
0808 06:57:31.715964 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.715966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.715973 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:31.715994 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.715995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.716006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.716070 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:31.716180 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.716674 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.717357 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.717567 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.717622 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.717759 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.717803 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.717857 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.717906 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.717933 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260808 06:57:31.721804 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.719869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:57937 (local address 127.21.28.65:40003)
0808 06:57:31.719997 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:31.719999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:31.720039 (+ 40us) server_negotiation.cc:407] Connection header received
0808 06:57:31.720086 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:31.720086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:31.720095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:31.720134 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:31.720211 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.720638 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.721273 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.721442 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.721509 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.721611 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.721634 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.721683 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:31.721714 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 06:57:31.721730 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260808 06:57:31.721817 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.719819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40003 (local address 127.21.28.65:57937)
0808 06:57:31.719979 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.719989 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:31.720050 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:31.720140 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:31.720141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:31.720142 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:57:31.720179 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.720180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.720682 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.720683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.721222 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:31.721223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:31.721511 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:31.721512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:31.721561 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:31.721565 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:57:31.721683 (+ 118us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:57:31.721684 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 06:57:31.721711 (+ 27us) client_negotiation.cc:240] Negotiation successful
0808 06:57:31.721745 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":109,"threads_started":1}
I20260808 06:57:31.723399 765 raft_consensus.cc:1261] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Refusing update from remote peer 9de4d11a44674988b2f2d5ef223b1b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.723397 551 raft_consensus.cc:1261] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Refusing update from remote peer 9de4d11a44674988b2f2d5ef223b1b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:31.723598 939 consensus_queue.cc:1035] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1f63b79076f480eb77a58c04b2ed08e" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 41887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.723721 952 consensus_queue.cc:1035] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e086e9a1d1d4791842bc5464e70e61b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:31.733711 32541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:31.733805 32541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:31.734342 32541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5cffb09f87e428a9491fc43d2e6c2e5: Sending DeleteTablet for 3 replicas of tablet 9851565a3e7b4ac89dbc0da0d36402f9
I20260808 06:57:31.734573 32749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9851565a3e7b4ac89dbc0da0d36402f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:60000
I20260808 06:57:31.734576 521 tablet_service.cc:1514] Processing DeleteTablet for tablet 9851565a3e7b4ac89dbc0da0d36402f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:38500
I20260808 06:57:31.734635 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 9851565a3e7b4ac89dbc0da0d36402f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:31 UTC) from {username='oryx', principal='oryx/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:39918
I20260808 06:57:31.734664 912 tablet_replica.cc:330] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a: stopping tablet replica
I20260808 06:57:31.734679 910 tablet_replica.cc:330] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: stopping tablet replica
I20260808 06:57:31.734707 912 raft_consensus.cc:2229] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:31.734730 910 raft_consensus.cc:2229] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:31.734728 908 tablet_replica.cc:330] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: stopping tablet replica
I20260808 06:57:31.734764 910 raft_consensus.cc:2258] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.734787 908 raft_consensus.cc:2229] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:31.734804 912 raft_consensus.cc:2258] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.734819 908 raft_consensus.cc:2258] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:31.735147 908 ts_tablet_manager.cc:1905] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.735148 912 ts_tablet_manager.cc:1905] T 9851565a3e7b4ac89dbc0da0d36402f9 P 9de4d11a44674988b2f2d5ef223b1b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.735204 910 ts_tablet_manager.cc:1905] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:31.736792 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 32652
I20260808 06:57:31.737418 910 ts_tablet_manager.cc:1918] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:31.737473 910 log.cc:1196] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/9851565a3e7b4ac89dbc0da0d36402f9
I20260808 06:57:31.737639 910 ts_tablet_manager.cc:1939] T 9851565a3e7b4ac89dbc0da0d36402f9 P 7e086e9a1d1d4791842bc5464e70e61b: Deleting consensus metadata
I20260808 06:57:31.737954 32522 catalog_manager.cc:4841] TS 7e086e9a1d1d4791842bc5464e70e61b (127.21.28.66:46341): tablet 9851565a3e7b4ac89dbc0da0d36402f9 (table test-table [id=74b1a12435dc4c0383c7b10d5bfc9ddc]) successfully deleted
I20260808 06:57:31.740862 908 ts_tablet_manager.cc:1918] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:57:31.740926 908 log.cc:1196] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/9851565a3e7b4ac89dbc0da0d36402f9
I20260808 06:57:31.741108 908 ts_tablet_manager.cc:1939] T 9851565a3e7b4ac89dbc0da0d36402f9 P a1f63b79076f480eb77a58c04b2ed08e: Deleting consensus metadata
I20260808 06:57:31.741420 32525 catalog_manager.cc:4841] TS a1f63b79076f480eb77a58c04b2ed08e (127.21.28.67:41887): tablet 9851565a3e7b4ac89dbc0da0d36402f9 (table test-table [id=74b1a12435dc4c0383c7b10d5bfc9ddc]) successfully deleted
W20260808 06:57:31.751770 32523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:40003 (error 108)
I20260808 06:57:31.751947 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 411
I20260808 06:57:31.761799 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:31.761600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:40003 (local address 127.0.0.1:60020)
0808 06:57:31.761715 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:31.761758 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:40003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":47,"threads_started":1}
W20260808 06:57:31.761919 32523 catalog_manager.cc:4584] TS 9de4d11a44674988b2f2d5ef223b1b6a (127.21.28.65:40003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9851565a3e7b4ac89dbc0da0d36402f9: Network error: Client connection negotiation failed: client connection to 127.21.28.65:40003: connect: Connection refused (error 111)
I20260808 06:57:31.766744 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 635
I20260808 06:57:31.780010 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 32501
2026-08-08T06:57:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2866 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:31 dist-test-slave-nj21 krb5kdc[1020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:31 dist-test-slave-nj21 krb5kdc[1020](info): set up 2 sockets
Aug 08 06:57:31 dist-test-slave-nj21 krb5kdc[1020](info): commencing operation
krb5kdc: starting...
W20260808 06:57:32.914498 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:32 dist-test-slave-nj21 krb5kdc[1020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:33.028563 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:42811
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:42811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:33.107116 1043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:33.107317 1043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:33.107347 1043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:33.108430 1043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:33.108464 1043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:33.108476 1043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:33.108487 1043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:33.108502 1043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:33.109901 1043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:42811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:33.110096 1043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:33.110271 1043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:33.114914 1055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:33.115021 1043 server_base.cc:1034] running on GCE node
W20260808 06:57:33.114928 1051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:33.114979 1053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:33.115386 1043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:33.115664 1043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:33.116825 1043 hybrid_clock.cc:648] HybridClock initialized: now 1786172253116810 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T06:57:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1337 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:33 dist-test-slave-nj21 krb5kdc[1061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:33 dist-test-slave-nj21 krb5kdc[1061](info): set up 2 sockets
Aug 08 06:57:33 dist-test-slave-nj21 krb5kdc[1061](info): commencing operation
krb5kdc: starting...
W20260808 06:57:34.257781 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:34.367627 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:40723
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:40723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:34.446645 1082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:34.446833 1082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:34.446854 1082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:34.447940 1082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:34.447973 1082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:34.447985 1082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:34.447996 1082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:34.448010 1082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:34.449463 1082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:40723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:40723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:34.449648 1082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:34.449812 1082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:34.454368 1089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:34.454430 1088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:34.454376 1091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:34.454552 1082 server_base.cc:1034] running on GCE node
I20260808 06:57:34.454807 1082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:34.455066 1082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:34.456203 1082 hybrid_clock.cc:648] HybridClock initialized: now 1786172254456193 us; error 33 us; skew 500 ppm
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:34.458463 1082 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:34.458868 1082 webserver.cc:460] Webserver started at http://127.21.28.126:45145/ using document root <none> and password file <none>
I20260808 06:57:34.459059 1082 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:34.459105 1082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:34.459224 1082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:34.460240 1082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "9aa13eb6877c4ed6b3d0776d7179402c"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "2908bef69d5be66fb8485c02a2f9560b"
server_key_iv: "d26c015086f0a3803c25e1d1cd651925"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.460563 1082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "9aa13eb6877c4ed6b3d0776d7179402c"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "2908bef69d5be66fb8485c02a2f9560b"
server_key_iv: "d26c015086f0a3803c25e1d1cd651925"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.461773 1082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:34.462411 1101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.462555 1082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:34.462639 1082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "9aa13eb6877c4ed6b3d0776d7179402c"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "2908bef69d5be66fb8485c02a2f9560b"
server_key_iv: "d26c015086f0a3803c25e1d1cd651925"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.462700 1082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:34.478204 1082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:34.479763 1082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:34.479889 1082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:34.483562 1082 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:40723
I20260808 06:57:34.483593 1178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:40723 every 8 connection(s)
I20260808 06:57:34.483974 1082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:34.484773 1179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:34.486964 1179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Bootstrap starting.
I20260808 06:57:34.487506 1179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:34.487800 1179 log.cc:824] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:34.488410 1179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: No bootstrap required, opened a new log
I20260808 06:57:34.489997 1179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } }
I20260808 06:57:34.490129 1179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:34.490160 1179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa13eb6877c4ed6b3d0776d7179402c, State: Initialized, Role: FOLLOWER
I20260808 06:57:34.490296 1179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } }
I20260808 06:57:34.490355 1179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:34.490387 1179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:34.490432 1179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:34.490931 1179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } }
I20260808 06:57:34.491021 1179 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aa13eb6877c4ed6b3d0776d7179402c; no voters:
I20260808 06:57:34.491174 1179 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:34.491210 1183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:34.491415 1179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:34.491503 1183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 1 LEADER]: Becoming Leader. State: Replica: 9aa13eb6877c4ed6b3d0776d7179402c, State: Running, Role: LEADER
I20260808 06:57:34.491609 1183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } }
I20260808 06:57:34.491719 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 1082
I20260808 06:57:34.491825 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:34.492033 21617 external_mini_cluster.cc:1428] Setting key 032294dcb771cc459262762888d37c21
I20260808 06:57:34.492358 1186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aa13eb6877c4ed6b3d0776d7179402c. Latest consensus state: current_term: 1 leader_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } } }
I20260808 06:57:34.492463 1186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:34.492512 1185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } } }
I20260808 06:57:34.492589 1185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:34.492909 1205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:34.493528 1205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:34.494792 1205 catalog_manager.cc:1347] Generated new cluster ID: f313a2568c6b412c8cea0cf61dc5efff
I20260808 06:57:34.494834 1205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:34.503492 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:34.492829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.21.28.126:40723)
0808 06:57:34.493071 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:34.493080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:34.493121 (+ 41us) server_negotiation.cc:407] Connection header received
0808 06:57:34.493230 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:34.493237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:34.493292 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:34.493447 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:34.494215 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.495561 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.496616 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.497231 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.497343 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:34.500074 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:34.500078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:34.500078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:34.500090 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:34.501952 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:34.502338 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:34.502340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:34.502341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:34.502442 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:34.502630 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:34.502631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:34.502631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:34.502908 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:34.503046 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:34.503145 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 06:57:34.503269 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":57,"threads_started":1}
I20260808 06:57:34.506601 1205 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:34.507057 1205 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:34.510524 1205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Generated new TSK 0
I20260808 06:57:34.510674 1205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:34.535987 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:40723
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:34.615003 1218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:34.615204 1218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:34.615235 1218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:34.616415 1218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:34.616457 1218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:34.616492 1218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:34.617887 1218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:34.618112 1218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:34.618323 1218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:34.623049 1226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:34.623059 1225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:34.623054 1231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:34.623155 1218 server_base.cc:1034] running on GCE node
I20260808 06:57:34.623437 1218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:34.623693 1218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:34.624837 1218 hybrid_clock.cc:648] HybridClock initialized: now 1786172254624825 us; error 29 us; skew 500 ppm
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:34.627193 1218 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:34.627578 1218 webserver.cc:460] Webserver started at http://127.21.28.65:46419/ using document root <none> and password file <none>
I20260808 06:57:34.627820 1218 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:34.627883 1218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:34.628031 1218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:34.629240 1218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "525b156bba4c41d596dcd110d3002759"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "266f602b3f86689521db30ea363c496e"
server_key_iv: "4d31d0cf95bbd2447a73ee127d512efd"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.629657 1218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "525b156bba4c41d596dcd110d3002759"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "266f602b3f86689521db30ea363c496e"
server_key_iv: "4d31d0cf95bbd2447a73ee127d512efd"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.631191 1218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:34.631836 1240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.631989 1218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:34.632076 1218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "525b156bba4c41d596dcd110d3002759"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "266f602b3f86689521db30ea363c496e"
server_key_iv: "4d31d0cf95bbd2447a73ee127d512efd"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.632143 1218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:34.646821 1218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:34.648816 1218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:34.648962 1218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:34.649161 1218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:34.649454 1218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:34.649487 1218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.649516 1218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:34.649539 1218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.655140 1218 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:37615
I20260808 06:57:34.655210 1404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:37615 every 8 connection(s)
I20260808 06:57:34.655467 1218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:34.660691 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 1218
I20260808 06:57:34.660804 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:34.661019 21617 external_mini_cluster.cc:1428] Setting key 0c454a0115ac42bf0bf11ac01c166344
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:34.663980 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:34.656128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:47285 (local address 127.21.28.126:40723)
0808 06:57:34.656209 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:34.656212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:34.656378 (+ 166us) server_negotiation.cc:407] Connection header received
0808 06:57:34.656829 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:34.656830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:34.656843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:34.656904 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:34.657759 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.658271 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.659177 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.659375 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.659427 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:34.661477 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:34.661479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:34.661479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:34.661485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:34.662698 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:34.663064 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:34.663065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:34.663066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:34.663122 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:34.663335 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:34.663336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:34.663336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:34.663452 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:34.663513 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:34.663797 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 06:57:34.663892 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:57:34.664126 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:34.656105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:47285)
0808 06:57:34.656265 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:34.656300 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:34.656756 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:34.656963 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:34.656966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:34.657508 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:34.657691 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:34.657696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.658332 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.658332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:34.659108 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:34.659109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.659439 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.659440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:34.659542 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:34.659978 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:34.659986 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:34.661402 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:34.662774 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:34.662781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:34.662783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:34.662996 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:34.663175 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:34.663176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:34.663176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:34.663279 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:34.663513 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:34.663520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:34.663650 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 06:57:34.663774 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 06:57:34.663923 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260808 06:57:34.664582 1405 heartbeater.cc:347] Connected to a master server at 127.21.28.126:40723
I20260808 06:57:34.664670 1405 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:34.664839 1405 heartbeater.cc:511] Master 127.21.28.126:40723 requested a full tablet report, sending...
I20260808 06:57:34.665364 1124 ts_manager.cc:194] Registered new tserver with Master: 525b156bba4c41d596dcd110d3002759 (127.21.28.65:37615)
I20260808 06:57:34.666278 1124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:47285
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:34.686060 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:40723
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:34.767010 1413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:34.767198 1413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:34.767220 1413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:34.768322 1413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:34.768358 1413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:34.768383 1413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:34.769765 1413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:34.769936 1413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:34.770109 1413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:34.774595 1431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:34.774695 1422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:34.774695 1424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:34.774814 1413 server_base.cc:1034] running on GCE node
I20260808 06:57:34.775035 1413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:34.775245 1413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:34.776383 1413 hybrid_clock.cc:648] HybridClock initialized: now 1786172254776371 us; error 25 us; skew 500 ppm
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:34.778750 1413 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:34.779134 1413 webserver.cc:460] Webserver started at http://127.21.28.66:39691/ using document root <none> and password file <none>
I20260808 06:57:34.779325 1413 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:34.779364 1413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:34.779454 1413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:34.780400 1413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "a30741f191154c3b8f365a4423fc4d69"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "230de2af84cc6e30843cc2031f00e72e"
server_key_iv: "ee92e89ed3a3bcc4ed538609fa06c920"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.780685 1413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "a30741f191154c3b8f365a4423fc4d69"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "230de2af84cc6e30843cc2031f00e72e"
server_key_iv: "ee92e89ed3a3bcc4ed538609fa06c920"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.782588 1413 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260808 06:57:34.783195 1442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.783332 1413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:34.783411 1413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "a30741f191154c3b8f365a4423fc4d69"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "230de2af84cc6e30843cc2031f00e72e"
server_key_iv: "ee92e89ed3a3bcc4ed538609fa06c920"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.783473 1413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:34.792174 1413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:34.793839 1413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:34.793958 1413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:34.794153 1413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:34.794431 1413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:34.794464 1413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.794485 1413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:34.794508 1413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.800201 1413 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:35385
I20260808 06:57:34.800269 1613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:35385 every 8 connection(s)
I20260808 06:57:34.800516 1413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:34.808810 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:34.801145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:46871 (local address 127.21.28.126:40723)
0808 06:57:34.801212 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:34.801214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:34.801364 (+ 150us) server_negotiation.cc:407] Connection header received
0808 06:57:34.801815 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:34.801816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:34.801826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:34.801875 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:34.802847 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.803362 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.804326 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.804518 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.804569 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:34.806489 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:34.806491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:34.806491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:34.806499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:34.807532 (+ 1033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:34.807938 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:34.807939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:34.807939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:34.807996 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:34.808210 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:34.808211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:34.808211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:34.808324 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:34.808380 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:34.808631 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 06:57:34.808721 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:57:34.808948 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:34.801099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:46871)
0808 06:57:34.801258 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:34.801296 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:34.801739 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:34.801931 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:34.801934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:34.802517 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:34.802742 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:34.802747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.803437 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.803438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:34.804262 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:34.804264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.804573 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.804574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:34.804665 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:34.805086 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:34.805094 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:34.806400 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:34.807605 (+ 1205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:34.807611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:34.807613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:34.807864 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:34.808047 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:34.808048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:34.808048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:34.808154 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:34.808378 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:34.808391 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:34.808516 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 06:57:34.808601 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 06:57:34.808748 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260808 06:57:34.809391 1614 heartbeater.cc:347] Connected to a master server at 127.21.28.126:40723
I20260808 06:57:34.809489 1614 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:34.809652 1614 heartbeater.cc:511] Master 127.21.28.126:40723 requested a full tablet report, sending...
I20260808 06:57:34.809996 1124 ts_manager.cc:194] Registered new tserver with Master: a30741f191154c3b8f365a4423fc4d69 (127.21.28.66:35385)
I20260808 06:57:34.810297 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 1413
I20260808 06:57:34.810329 1124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:46871
I20260808 06:57:34.810387 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:34.810591 21617 external_mini_cluster.cc:1428] Setting key 0927c885aee6441aae16e829352acd04
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:34.834326 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:40723
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:34.913085 1621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:34.913277 1621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:34.913307 1621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:34.914424 1621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:34.914462 1621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:34.914495 1621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:34.915879 1621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:34.916091 1621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:34.916294 1621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:34.920964 1633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:34.921008 1621 server_base.cc:1034] running on GCE node
W20260808 06:57:34.920956 1637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:34.920956 1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:34.921329 1621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:34.921566 1621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:34.922703 1621 hybrid_clock.cc:648] HybridClock initialized: now 1786172254922691 us; error 30 us; skew 500 ppm
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:34.925145 1621 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:34.925565 1621 webserver.cc:460] Webserver started at http://127.21.28.67:38417/ using document root <none> and password file <none>
I20260808 06:57:34.925761 1621 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:34.925806 1621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:34.925930 1621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:34.926890 1621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "b18515a394524947be9a30c729a9a541"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "0bc6793e29c8b7953bf82bb3ef695d85"
server_key_iv: "539f2d8b31e6330f9f36e6fe34b04c09"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.927213 1621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "b18515a394524947be9a30c729a9a541"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "0bc6793e29c8b7953bf82bb3ef695d85"
server_key_iv: "539f2d8b31e6330f9f36e6fe34b04c09"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.928481 1621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 06:57:34.929158 1647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.929317 1621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:34.929415 1621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "b18515a394524947be9a30c729a9a541"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "0bc6793e29c8b7953bf82bb3ef695d85"
server_key_iv: "539f2d8b31e6330f9f36e6fe34b04c09"
server_key_version: "encryptionkey@0"
I20260808 06:57:34.929486 1621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:34.936374 1621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:34.938235 1621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:34.938364 1621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:34.938560 1621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:34.938813 1621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:34.938844 1621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.938869 1621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:34.938894 1621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:34.944674 1621 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:43699
I20260808 06:57:34.944728 1822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:43699 every 8 connection(s)
I20260808 06:57:34.944986 1621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:34.948587 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 1621
I20260808 06:57:34.948700 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:34.948879 21617 external_mini_cluster.cc:1428] Setting key 21ec531403e29dbf11d20199c54377af
Aug 08 06:57:34 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172254, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:34.952782 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:34.945599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:54961 (local address 127.21.28.126:40723)
0808 06:57:34.945648 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:34.945650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:34.945835 (+ 185us) server_negotiation.cc:407] Connection header received
0808 06:57:34.946267 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:34.946269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:34.946279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:34.946322 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:34.947139 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.947623 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.948549 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.948828 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.948897 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:34.950738 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:34.950740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:34.950740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:34.950746 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:34.951808 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:34.952117 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:34.952117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:34.952118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:34.952165 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:34.952314 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:34.952314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:34.952315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:34.952428 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:34.952474 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:34.952660 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 06:57:34.952715 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:34.952934 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:34.945560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:54961)
0808 06:57:34.945716 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:34.945749 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:34.946195 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:34.946378 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:34.946381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:34.946898 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:34.947076 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:34.947080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.947707 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.947708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:34.948477 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:34.948479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:34.948900 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:34.948901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:34.949022 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:34.949431 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:34.949438 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:34.950663 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:34.951867 (+ 1204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:34.951872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:34.951873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:34.952064 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:34.952200 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:34.952201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:34.952201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:34.952277 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:34.952471 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:34.952474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:34.952570 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 06:57:34.952643 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 06:57:34.952752 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260808 06:57:34.953367 1824 heartbeater.cc:347] Connected to a master server at 127.21.28.126:40723
I20260808 06:57:34.953459 1824 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:34.953616 1824 heartbeater.cc:511] Master 127.21.28.126:40723 requested a full tablet report, sending...
I20260808 06:57:34.953948 1124 ts_manager.cc:194] Registered new tserver with Master: b18515a394524947be9a30c729a9a541 (127.21.28.67:43699)
I20260808 06:57:34.954301 1124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:54961
I20260808 06:57:34.960289 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab.
Entry for principal oryx/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab.
WARNING: no policy specified for oryx/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab.
Entry for principal oryx/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab.
WARNING: no policy specified for oryx/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab.
Entry for principal oryx/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab.
I20260808 06:57:35.036422 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 1218
I20260808 06:57:35.049615 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 1413
I20260808 06:57:35.068195 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 1621
I20260808 06:57:35.080482 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 1082
I20260808 06:57:35.092768 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:40723
--webserver_interface=127.21.28.126
--webserver_port=45145
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:40723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:35.173959 1835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:35.174141 1835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:35.174161 1835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:35.175264 1835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:35.175304 1835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:35.175316 1835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:35.175340 1835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:35.175362 1835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:35.176813 1835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:40723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:40723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=45145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:35.177050 1835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:35.177250 1835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:35.181840 1842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:35.181829 1847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:35.181857 1843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:35.182103 1835 server_base.cc:1034] running on GCE node
I20260808 06:57:35.182317 1835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:35.182577 1835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:35.183723 1835 hybrid_clock.cc:648] HybridClock initialized: now 1786172255183704 us; error 34 us; skew 500 ppm
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172255, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:35.186163 1835 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:35.186560 1835 webserver.cc:460] Webserver started at http://127.21.28.126:45145/ using document root <none> and password file <none>
I20260808 06:57:35.186746 1835 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:35.186805 1835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:35.188100 1835 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:35.188663 1854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.188812 1835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:35.188910 1835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "9aa13eb6877c4ed6b3d0776d7179402c"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "2908bef69d5be66fb8485c02a2f9560b"
server_key_iv: "d26c015086f0a3803c25e1d1cd651925"
server_key_version: "encryptionkey@0"
I20260808 06:57:35.189205 1835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:35.202036 1835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:35.204110 1835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:35.204236 1835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:35.208011 1835 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:40723
I20260808 06:57:35.208043 1932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:40723 every 8 connection(s)
I20260808 06:57:35.208388 1835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:35.209762 1933 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:57:35.210415 1933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Bootstrap starting.
I20260808 06:57:35.217263 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 1835
I20260808 06:57:35.217605 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:37615
--local_ip_for_outbound_sockets=127.21.28.65
--tserver_master_addrs=127.21.28.126:40723
--webserver_port=46419
--webserver_interface=127.21.28.65
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.28.65
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:57:35.219120 1933 log.cc:824] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:35.220784 1933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:57:35.221040 1933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Bootstrap complete.
I20260808 06:57:35.223134 1933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } }
I20260808 06:57:35.223356 1933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa13eb6877c4ed6b3d0776d7179402c, State: Initialized, Role: FOLLOWER
I20260808 06:57:35.223503 1933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } }
I20260808 06:57:35.223569 1933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:35.223609 1933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:35.223650 1933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:57:35.224763 1933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } }
I20260808 06:57:35.224854 1933 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aa13eb6877c4ed6b3d0776d7179402c; no voters:
I20260808 06:57:35.225008 1933 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 06:57:35.225052 1941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:57:35.225174 1941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [term 2 LEADER]: Becoming Leader. State: Replica: 9aa13eb6877c4ed6b3d0776d7179402c, State: Running, Role: LEADER
I20260808 06:57:35.225268 1933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:35.225271 1941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } }
I20260808 06:57:35.225611 1941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aa13eb6877c4ed6b3d0776d7179402c. Latest consensus state: current_term: 2 leader_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } } }
I20260808 06:57:35.225695 1941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:35.225903 1943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa13eb6877c4ed6b3d0776d7179402c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40723 } } }
I20260808 06:57:35.226016 1943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:35.226166 1949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:35.226670 1949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:35.226806 1949 catalog_manager.cc:1259] Loaded cluster ID: f313a2568c6b412c8cea0cf61dc5efff
I20260808 06:57:35.227007 1949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:35.228222 1949 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:35.228410 1949 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9aa13eb6877c4ed6b3d0776d7179402c: Loaded TSK: 0
I20260808 06:57:35.228623 1949 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:57:35.298710 1936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:35.298914 1936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:35.298949 1936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:35.300110 1936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:35.300149 1936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:35.300185 1936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:35.301584 1936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.65.keytab
--principal=oryx/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:37615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=46419
--tserver_master_addrs=127.21.28.126:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:35.301800 1936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:35.302004 1936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:35.306890 1974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:35.307017 1936 server_base.cc:1034] running on GCE node
W20260808 06:57:35.306923 1975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:35.307019 1977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:35.307346 1936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:35.307578 1936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:35.308725 1936 hybrid_clock.cc:648] HybridClock initialized: now 1786172255308715 us; error 27 us; skew 500 ppm
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172255, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:35.311225 1936 init.cc:377] Logged in from keytab as oryx/127.21.28.65@KRBTEST.COM (short username oryx)
I20260808 06:57:35.311653 1936 webserver.cc:460] Webserver started at http://127.21.28.65:46419/ using document root <none> and password file <none>
I20260808 06:57:35.311910 1936 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:35.311975 1936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:35.313374 1936 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:35.313987 1994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.314154 1936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:57:35.314249 1936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "525b156bba4c41d596dcd110d3002759"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "266f602b3f86689521db30ea363c496e"
server_key_iv: "4d31d0cf95bbd2447a73ee127d512efd"
server_key_version: "encryptionkey@0"
I20260808 06:57:35.314692 1936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:35.341252 1936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:35.343645 1936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:35.343847 1936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:35.344136 1936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:35.344511 1936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:35.344556 1936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.344590 1936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:35.344612 1936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.351867 1936 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:37615
I20260808 06:57:35.351920 2175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:37615 every 8 connection(s)
I20260808 06:57:35.352283 1936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:35.353076 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 1936
I20260808 06:57:35.353389 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:35385
--local_ip_for_outbound_sockets=127.21.28.66
--tserver_master_addrs=127.21.28.126:40723
--webserver_port=39691
--webserver_interface=127.21.28.66
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.28.66
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:35.361368 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.352988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:45481)
0808 06:57:35.353343 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.353376 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.353955 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.354371 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.354375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.355035 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.355311 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.355318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.356842 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.356843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.357869 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.357872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.358445 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.358446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.358574 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.359050 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.359058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.361229 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":94,"threads_started":1}
W20260808 06:57:35.361562 2176 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:40723 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.28.126:40723: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
W20260808 06:57:35.361578 2181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.353028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:45481 (local address 127.21.28.126:40723)
0808 06:57:35.353270 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.353278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.353470 (+ 192us) server_negotiation.cc:407] Connection header received
0808 06:57:35.354160 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.354165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.354206 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.354317 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.355449 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.356714 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.357963 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.358360 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.358460 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.361406 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:45481: BlockingRecv error: recv got EOF from 127.21.28.65:45481 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:35.366039 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.361816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:60421)
0808 06:57:35.361907 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.361916 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.361961 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.362095 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.362095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.362204 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.362247 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.362249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.363082 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.363082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.363548 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.363549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.363993 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.363993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.364044 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.364420 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.364423 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.365997 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260808 06:57:35.366147 2181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.361861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:60421 (local address 127.21.28.126:40723)
0808 06:57:35.361921 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.361923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.361956 (+ 33us) server_negotiation.cc:407] Connection header received
0808 06:57:35.362023 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.362024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.362035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.362084 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.362307 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.363009 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.363615 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.363919 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.363986 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.366095 (+ 2109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:60421: BlockingRecv error: recv got EOF from 127.21.28.65:60421 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:35.370247 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.366249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:58757)
0808 06:57:35.366281 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.366292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.366329 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.366468 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.366469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.366604 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.366635 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.366636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.367177 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.367177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.368018 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.368020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.368318 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.368318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.368369 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.368682 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.368687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.370209 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260808 06:57:35.370345 2176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:57:35.370358 2181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.366333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:58757 (local address 127.21.28.126:40723)
0808 06:57:35.366369 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.366370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.366378 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:35.366397 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.366398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.366408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.366452 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.366691 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.367120 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.368086 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.368258 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.368307 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.370306 (+ 1999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:58757: BlockingRecv error: recv got EOF from 127.21.28.65:58757 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 06:57:35.438047 2182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:35.438239 2182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:35.438261 2182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:35.439356 2182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:35.439391 2182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:35.439416 2182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:35.440860 2182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.66.keytab
--principal=oryx/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:35385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=39691
--tserver_master_addrs=127.21.28.126:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:35.441035 2182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:35.441210 2182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:35.445982 2190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:35.445974 2191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:35.445972 2193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:35.446223 2182 server_base.cc:1034] running on GCE node
I20260808 06:57:35.446421 2182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:35.446638 2182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:35.447764 2182 hybrid_clock.cc:648] HybridClock initialized: now 1786172255447748 us; error 28 us; skew 500 ppm
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172255, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:35.450071 2182 init.cc:377] Logged in from keytab as oryx/127.21.28.66@KRBTEST.COM (short username oryx)
I20260808 06:57:35.450496 2182 webserver.cc:460] Webserver started at http://127.21.28.66:39691/ using document root <none> and password file <none>
I20260808 06:57:35.450687 2182 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:35.450749 2182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:35.452044 2182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:35.452555 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.452708 2182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.452790 2182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "a30741f191154c3b8f365a4423fc4d69"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "230de2af84cc6e30843cc2031f00e72e"
server_key_iv: "ee92e89ed3a3bcc4ed538609fa06c920"
server_key_version: "encryptionkey@0"
I20260808 06:57:35.453083 2182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:35.465139 2182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:35.466777 2182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:35.466903 2182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:35.467101 2182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:35.467371 2182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:35.467406 2182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.467432 2182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:35.467449 2182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.473292 2182 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:35385
I20260808 06:57:35.473356 2372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:35385 every 8 connection(s)
I20260808 06:57:35.473644 2182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:35.478734 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 2182
I20260808 06:57:35.479003 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:43699
--local_ip_for_outbound_sockets=127.21.28.67
--tserver_master_addrs=127.21.28.126:40723
--webserver_port=38417
--webserver_interface=127.21.28.67
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.28.67
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:57:35.480636 2181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.474285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:44525 (local address 127.21.28.126:40723)
0808 06:57:35.474342 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.474344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.474547 (+ 203us) server_negotiation.cc:407] Connection header received
0808 06:57:35.475024 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.475025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.475042 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.475093 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.476009 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.476517 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.477449 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.477651 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.477706 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.480578 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:44525: BlockingRecv error: recv got EOF from 127.21.28.66:44525 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 06:57:35.480661 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.474258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:44525)
0808 06:57:35.474430 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.474462 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.474927 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.475159 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.475163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.475725 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.475924 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.475929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.476598 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.476599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.477365 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.477367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.477718 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.477719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.477828 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.478225 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.478232 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.480477 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
W20260808 06:57:35.480937 2373 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:40723 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.28.126:40723: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:57:35.485248 2181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.481185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:48073 (local address 127.21.28.126:40723)
0808 06:57:35.481243 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.481245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.481254 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:35.481276 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.481276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.481285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.481332 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.481553 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.482009 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.482631 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.482833 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.482885 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.485192 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:48073: BlockingRecv error: recv got EOF from 127.21.28.66:48073 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 06:57:35.485400 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.481127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:48073)
0808 06:57:35.481167 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.481177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.481222 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.481333 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.481334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.481452 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.481491 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.481492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.482073 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.482074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.482564 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.482565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.482888 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.482889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.482956 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.483298 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.483302 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.485336 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:35.489605 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.485674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:44189)
0808 06:57:35.485714 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.485725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.485761 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.485892 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.485893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.485990 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.486015 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.486015 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.486827 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.486828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.487299 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.487300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.487748 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.487748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.487786 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.488106 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.488111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.489564 (+ 1453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 06:57:35.489702 2373 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:57:35.489710 2181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.485742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:44189 (local address 127.21.28.126:40723)
0808 06:57:35.485781 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.485783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.485792 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:35.485813 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.485813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.485822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.485889 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.486072 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.486744 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.487371 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.487655 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.487750 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.489652 (+ 1902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:44189: BlockingRecv error: recv got EOF from 127.21.28.66:44189 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 06:57:35.560144 2376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:35.560335 2376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:35.560357 2376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:35.561506 2376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:35.561545 2376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:35.561573 2376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:35.562968 2376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/krb5kdc/oryx_127.21.28.67.keytab
--principal=oryx/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:43699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=38417
--tserver_master_addrs=127.21.28.126:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:35.563179 2376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:35.563380 2376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:57:35.568117 2376 server_base.cc:1034] running on GCE node
W20260808 06:57:35.568094 2385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:35.568102 2386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:35.568102 2389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:35.568470 2376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:35.568693 2376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:35.569842 2376 hybrid_clock.cc:648] HybridClock initialized: now 1786172255569828 us; error 27 us; skew 500 ppm
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172255, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:35.572232 2376 init.cc:377] Logged in from keytab as oryx/127.21.28.67@KRBTEST.COM (short username oryx)
I20260808 06:57:35.572599 2376 webserver.cc:460] Webserver started at http://127.21.28.67:38417/ using document root <none> and password file <none>
I20260808 06:57:35.572799 2376 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:35.572853 2376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:35.574084 2376 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:35.574569 2400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.574709 2376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.574781 2376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "b18515a394524947be9a30c729a9a541"
format_stamp: "Formatted at 2026-08-08 06:57:34 on dist-test-slave-nj21"
server_key: "0bc6793e29c8b7953bf82bb3ef695d85"
server_key_iv: "539f2d8b31e6330f9f36e6fe34b04c09"
server_key_version: "encryptionkey@0"
I20260808 06:57:35.575083 2376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:35.591969 2376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:35.593716 2376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:35.593827 2376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:35.594049 2376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:35.594301 2376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:35.594331 2376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.594350 2376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:35.594367 2376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:35.599996 2376 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:43699
I20260808 06:57:35.600037 2574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:43699 every 8 connection(s)
I20260808 06:57:35.600342 2376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:35.604111 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 2376
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:57:35.607635 2181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.601068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:33807 (local address 127.21.28.126:40723)
0808 06:57:35.601131 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.601135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.601344 (+ 209us) server_negotiation.cc:407] Connection header received
0808 06:57:35.601810 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.601811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.601821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.601870 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.602723 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.603225 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.604200 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.604412 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.604465 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.607585 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:33807: BlockingRecv error: recv got EOF from 127.21.28.67:33807 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 06:57:35.607647 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.601059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:33807)
0808 06:57:35.601230 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.601264 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.601733 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.601923 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.601926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.602472 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.602654 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.602659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.603286 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.603287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.604108 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.604110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.604495 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.604496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.604637 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.605124 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.605135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.607486 (+ 2351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260808 06:57:35.607899 2575 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:40723 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.28.126:40723: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:35.612185 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.608114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:44949)
0808 06:57:35.608148 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.608158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.608192 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.608339 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.608340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.608483 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.608541 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.608543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.609113 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.609114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.609683 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.609685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.610023 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.610023 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.610091 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.610391 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.610394 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.612127 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260808 06:57:35.612291 2181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.608175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:44949 (local address 127.21.28.126:40723)
0808 06:57:35.608230 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.608232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.608239 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:57:35.608258 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.608259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.608267 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.608318 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.608616 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.609041 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.609766 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.609951 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.610005 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.612231 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:44949: BlockingRecv error: recv got EOF from 127.21.28.67:44949 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:57:35.612398 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.604592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.21.28.126:40723)
0808 06:57:35.604672 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.604673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.604713 (+ 40us) server_negotiation.cc:407] Connection header received
0808 06:57:35.604798 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.604798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.604806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.604867 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.605621 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.606265 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.607128 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.607415 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.607488 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.609291 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:35.609295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:35.609295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:35.609305 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:35.610859 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:35.611240 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:35.611242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:35.611242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:35.611340 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:35.611540 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:35.611541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:35.611541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:35.611911 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:35.612023 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:35.612180 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 06:57:35.612295 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":40,"threads_started":1}
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:35 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:35.616501 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:35.612936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:58329)
0808 06:57:35.612971 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:35.612981 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:35.613022 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:35.613162 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:35.613163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:35.613292 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:35.613323 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.613324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.613899 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.613899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.614358 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:35.614359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.614625 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.614626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:35.614657 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.614972 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:35.614977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:35.616460 (+ 1483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260808 06:57:35.616614 2583 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:35.613028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:58329 (local address 127.21.28.126:40723)
0808 06:57:35.613062 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:35.613064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:35.613072 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:35.613091 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:35.613092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:35.613103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:35.613159 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:35.613384 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.613836 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.614409 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:35.614577 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:35.614623 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:35.616561 (+ 1938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:58329: BlockingRecv error: recv got EOF from 127.21.28.67:58329 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 06:57:35.616647 2575 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 06:57:36 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:36 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:36.379266 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:36.370724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:50101)
0808 06:57:36.370858 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:36.370869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:36.370929 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:36.371062 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:36.371063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:36.371222 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:36.371275 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:36.371277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.371897 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.371898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:36.372471 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:36.372472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.372765 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.372765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:36.372801 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:36.377484 (+ 4683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:36.377490 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:36.379178 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260808 06:57:36.379376 2588 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:36.370810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:50101 (local address 127.21.28.126:40723)
0808 06:57:36.370925 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:36.370927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:36.370936 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:36.370975 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:36.370976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:36.370984 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:36.371038 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:36.371329 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.371846 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.372531 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.372718 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.372760 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:36.379302 (+ 6542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:50101: BlockingRecv error: recv got EOF from 127.21.28.65:50101 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
Aug 08 06:57:36 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:36 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:36.494315 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:36.490027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:56501)
0808 06:57:36.490154 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:36.490166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:36.490239 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:36.490364 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:36.490364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:36.490540 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:36.490590 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:36.490592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.491228 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.491229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:36.491805 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:36.491808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.492140 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.492141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:36.492197 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:36.492585 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:36.492589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:36.494231 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20260808 06:57:36.494388 2588 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:36.490101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:56501 (local address 127.21.28.126:40723)
0808 06:57:36.490158 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:36.490160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:36.490230 (+ 70us) server_negotiation.cc:407] Connection header received
0808 06:57:36.490290 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:36.490291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:36.490302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:36.490350 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:36.490664 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.491152 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.491884 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.492075 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.492130 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:36.494327 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:56501: BlockingRecv error: recv got EOF from 127.21.28.66:56501 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 08 06:57:36 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:36 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:36.621309 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:36.616978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:45625)
0808 06:57:36.617108 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:36.617118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:36.617183 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:36.617312 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:36.617313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:36.617491 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:36.617538 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:36.617541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.618205 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.618206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:36.618738 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:36.618740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.619109 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.619110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:36.619167 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:36.619595 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:36.619600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:36.621227 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
W20260808 06:57:36.621377 2588 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:36.617024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:45625 (local address 127.21.28.126:40723)
0808 06:57:36.617068 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:36.617071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:36.617171 (+ 100us) server_negotiation.cc:407] Connection header received
0808 06:57:36.617240 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:36.617241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:36.617254 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:36.617301 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:36.617604 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.618126 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.618809 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:36.619045 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:36.619098 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:36.621323 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:45625: BlockingRecv error: recv got EOF from 127.21.28.67:45625 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 08 06:57:37 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:37 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:37.383972 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:37.379796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:37003)
0808 06:57:37.379919 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:37.379932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:37.379990 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:37.380108 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:37.380109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:37.380284 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:37.380340 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:37.380342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.380952 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.380952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:37.381491 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:37.381493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.381766 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.381766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:37.381817 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:37.382264 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:37.382268 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:37.383913 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
W20260808 06:57:37.384094 2593 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:37.379856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:37003 (local address 127.21.28.126:40723)
0808 06:57:37.379977 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:37.379979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:37.379987 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:37.380034 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:37.380035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:37.380044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:37.380097 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:37.380396 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.380897 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.381545 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.381722 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.381762 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:37.384022 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:37003: BlockingRecv error: recv got EOF from 127.21.28.65:37003 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
Aug 08 06:57:37 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:37 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:37.499177 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:37.494786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:47493)
0808 06:57:37.494914 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:37.494924 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:37.494986 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:37.495106 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:37.495107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:37.495270 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:37.495322 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:37.495324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.495967 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.495968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:37.496674 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:37.496677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.496976 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.496976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:37.497019 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:37.497458 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:37.497463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:37.499100 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
W20260808 06:57:37.499255 2593 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:37.494855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:47493 (local address 127.21.28.126:40723)
0808 06:57:37.494914 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:37.494916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:37.494983 (+ 67us) server_negotiation.cc:407] Connection header received
0808 06:57:37.495043 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:37.495044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:37.495055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:37.495093 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:37.495371 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.495904 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.496736 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.496923 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.496973 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:37.499201 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:47493: BlockingRecv error: recv got EOF from 127.21.28.66:47493 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 08 06:57:37 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:37 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:37.626199 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:37.621801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:57917)
0808 06:57:37.621963 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:37.621972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:37.622016 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:37.622144 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:37.622145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:37.622313 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:37.622364 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:37.622366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.622968 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.622968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:37.623507 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:37.623509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.623823 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.623823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:37.623862 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:37.624279 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:37.624284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:37.626122 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
W20260808 06:57:37.626297 2593 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:37.621949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:57917 (local address 127.21.28.126:40723)
0808 06:57:37.622006 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:37.622008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:37.622017 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:37.622076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:37.622077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:37.622089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:37.622130 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:37.622425 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.622914 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.623564 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:37.623769 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:37.623814 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:37.626224 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:57917: BlockingRecv error: recv got EOF from 127.21.28.67:57917 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 08 06:57:38 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:38 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:38.388805 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:38.384427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:45877)
0808 06:57:38.384565 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:38.384576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:38.384639 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:38.384773 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:38.384774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:38.384941 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:38.384996 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:38.384998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.385596 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.385596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:38.386185 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:38.386188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.386558 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.386559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:38.386616 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:38.387015 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:38.387019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:38.388712 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
W20260808 06:57:38.388872 2599 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:38.384488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:45877 (local address 127.21.28.126:40723)
0808 06:57:38.384592 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:38.384594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:38.384622 (+ 28us) server_negotiation.cc:407] Connection header received
0808 06:57:38.384694 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:38.384695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:38.384705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:38.384762 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:38.385045 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.385546 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.386250 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.386499 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.386560 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:38.388810 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:45877: BlockingRecv error: recv got EOF from 127.21.28.65:45877 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
Aug 08 06:57:38 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:38 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:38.503954 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:38.499620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:44757)
0808 06:57:38.499794 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:38.499805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:38.499875 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:38.500000 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:38.500000 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:38.500168 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:38.500222 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:38.500224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.500824 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.500825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:38.501342 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:38.501343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.501619 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.501620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:38.501655 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:38.502058 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:38.502062 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:38.503869 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
W20260808 06:57:38.504029 2599 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:38.499717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:44757 (local address 127.21.28.126:40723)
0808 06:57:38.499772 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:38.499774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:38.499871 (+ 97us) server_negotiation.cc:407] Connection header received
0808 06:57:38.499929 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:38.499930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:38.499940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:38.499985 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:38.500276 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.500770 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.501394 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.501572 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.501615 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:38.503966 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:44757: BlockingRecv error: recv got EOF from 127.21.28.66:44757 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 08 06:57:38 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:38 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:38.635357 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:38.626657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:53265)
0808 06:57:38.626806 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:38.626818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:38.626872 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:38.627018 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:38.627018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:38.627180 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:38.627230 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:38.627232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.628006 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.628007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:38.628540 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:38.628542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.628913 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.628913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:38.628962 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:38.633579 (+ 4617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:38.633584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:38.635296 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
W20260808 06:57:38.635448 2599 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:38.626791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:53265 (local address 127.21.28.126:40723)
0808 06:57:38.626855 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:38.626857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:38.626867 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:38.626938 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:38.626939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:38.626953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:38.626999 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:38.627330 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.627908 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.628621 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:38.628835 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:38.628889 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:38.635381 (+ 6492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:53265: BlockingRecv error: recv got EOF from 127.21.28.67:53265 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 06:57:39 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:39 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:39.393549 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:39.389241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:47853)
0808 06:57:39.389376 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:39.389388 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:39.389455 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:39.389577 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:39.389577 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:39.389735 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:39.389789 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:39.389790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.390385 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.390386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:39.390915 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:39.390916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.391205 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.391206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:39.391251 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:39.391710 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:39.391714 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:39.393489 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
W20260808 06:57:39.393662 2604 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:39.389280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:47853 (local address 127.21.28.126:40723)
0808 06:57:39.389393 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:39.389395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:39.389429 (+ 34us) server_negotiation.cc:407] Connection header received
0808 06:57:39.389506 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:39.389507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:39.389517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:39.389564 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:39.389837 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.390330 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.390967 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.391154 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.391196 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:39.393585 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:47853: BlockingRecv error: recv got EOF from 127.21.28.65:47853 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
Aug 08 06:57:39 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:39 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:39.508713 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:39.504380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:50667)
0808 06:57:39.504532 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:39.504543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:39.504614 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:39.504734 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:39.504735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:39.504924 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:39.504972 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:39.504974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.505605 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.505606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:39.506130 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:39.506132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.506445 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.506446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:39.506499 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:39.506926 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:39.506930 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:39.508652 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
W20260808 06:57:39.508796 2604 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:39.504451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:50667 (local address 127.21.28.126:40723)
0808 06:57:39.504513 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:39.504515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:39.504602 (+ 87us) server_negotiation.cc:407] Connection header received
0808 06:57:39.504662 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:39.504662 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:39.504674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:39.504723 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:39.505039 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.505535 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.506199 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.506388 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.506447 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:39.508738 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:50667: BlockingRecv error: recv got EOF from 127.21.28.66:50667 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 08 06:57:39 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:39 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:39.640134 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:39.635834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:60421)
0808 06:57:39.635973 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:39.635985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:39.636048 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:39.636173 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:39.636173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:39.636350 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:39.636398 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:39.636400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.637031 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.637032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:39.637540 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:39.637542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.637864 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.637864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:39.637929 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:39.638355 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:39.638359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:39.640051 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
W20260808 06:57:39.640223 2604 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:39.635926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:60421 (local address 127.21.28.126:40723)
0808 06:57:39.635978 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:39.635980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:39.636030 (+ 50us) server_negotiation.cc:407] Connection header received
0808 06:57:39.636097 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:39.636098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:39.636111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:39.636159 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:39.636471 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.636958 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.637619 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:39.637803 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:39.637855 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:39.640166 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:60421: BlockingRecv error: recv got EOF from 127.21.28.67:60421 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 08 06:57:40 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:40 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:40.398099 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:40.393966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:45497)
0808 06:57:40.394097 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:40.394107 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:40.394159 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:40.394282 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:40.394283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:40.394447 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:40.394500 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:40.394501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.395119 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.395119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:40.395652 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:40.395654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.395968 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.395969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:40.396004 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:40.396415 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:40.396419 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:40.398039 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
W20260808 06:57:40.398202 2608 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:40.394023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:45497 (local address 127.21.28.126:40723)
0808 06:57:40.394128 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:40.394130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:40.394142 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:57:40.394208 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:40.394209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:40.394218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:40.394271 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:40.394550 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.395063 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.395721 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.395921 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.395965 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:40.398133 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:45497: BlockingRecv error: recv got EOF from 127.21.28.65:45497 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
Aug 08 06:57:40 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:40 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:40.513370 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:40.509164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:42543)
0808 06:57:40.509308 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:40.509320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:40.509388 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:40.509526 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:40.509526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:40.509690 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:40.509741 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:40.509743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.510343 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.510344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:40.510879 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:40.510881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.511158 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.511158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:40.511193 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:40.511604 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:40.511608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:40.513311 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
W20260808 06:57:40.513494 2608 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:40.509270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:42543 (local address 127.21.28.126:40723)
0808 06:57:40.509330 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:40.509332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:40.509392 (+ 60us) server_negotiation.cc:407] Connection header received
0808 06:57:40.509455 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:40.509456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:40.509469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:40.509514 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:40.509792 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.510291 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.510933 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.511113 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.511156 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:40.513424 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:42543: BlockingRecv error: recv got EOF from 127.21.28.66:42543 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 08 06:57:40 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:40 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:40.644980 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:40.640610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:58493)
0808 06:57:40.640745 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:40.640757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:40.640831 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:40.640978 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:40.640979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:40.641156 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:40.641202 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:40.641205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.641849 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.641850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:40.642360 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:40.642362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.642685 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.642687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:40.642743 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:40.643169 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:40.643173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:40.644918 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
W20260808 06:57:40.645071 2608 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:40.640671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:58493 (local address 127.21.28.126:40723)
0808 06:57:40.640747 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:40.640749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:40.640822 (+ 73us) server_negotiation.cc:407] Connection header received
0808 06:57:40.640892 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:40.640893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:40.640913 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:40.640964 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:40.641274 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.641776 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.642438 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:40.642626 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:40.642680 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:40.645009 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:58493: BlockingRecv error: recv got EOF from 127.21.28.67:58493 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 06:57:41 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:41 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:41.405431 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:41.398528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:36217)
0808 06:57:41.398651 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:41.398662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:41.398711 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:41.398830 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:41.398831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:41.398993 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:41.399062 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:41.399065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.399658 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.399659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:41.400205 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:41.400206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.400497 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.400498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:41.400545 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:41.403712 (+ 3167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:41.403716 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:41.405368 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260808 06:57:41.405530 2614 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:41.398585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:36217 (local address 127.21.28.126:40723)
0808 06:57:41.398699 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:41.398701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:41.398709 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:41.398752 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:41.398753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:41.398763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:41.398820 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:41.399113 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.399606 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.400253 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.400450 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.400495 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:41.405463 (+ 4968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:36217: BlockingRecv error: recv got EOF from 127.21.28.65:36217 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
Aug 08 06:57:41 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:41 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:41.518131 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:41.513828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:57159)
0808 06:57:41.514027 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:41.514038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:41.514098 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:41.514227 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:41.514228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:41.514397 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:41.514461 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:41.514464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.515108 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.515109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:41.515616 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:41.515617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.515967 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.515968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:41.516034 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:41.516419 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:41.516423 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:41.518031 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
W20260808 06:57:41.518181 2614 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:41.513902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:57159 (local address 127.21.28.126:40723)
0808 06:57:41.513958 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:41.513960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:41.514104 (+ 144us) server_negotiation.cc:407] Connection header received
0808 06:57:41.514156 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:41.514157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:41.514169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:41.514215 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:41.514536 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.515034 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.515706 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.515904 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.515957 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:41.518128 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:57159: BlockingRecv error: recv got EOF from 127.21.28.66:57159 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 08 06:57:41 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:41 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:41.650247 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:41.645394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:34397)
0808 06:57:41.645530 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:41.645543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:41.645610 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:41.645739 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:41.645740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:41.645930 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:41.645997 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:41.646000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.646653 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.646654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:41.647196 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:41.647198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.647528 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.647528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:41.647604 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:41.648025 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:41.648030 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:41.650164 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
W20260808 06:57:41.650367 2614 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:41.645451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:34397 (local address 127.21.28.126:40723)
0808 06:57:41.645534 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:41.645536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:41.645601 (+ 65us) server_negotiation.cc:407] Connection header received
0808 06:57:41.645669 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:41.645669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:41.645684 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:41.645732 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:41.646071 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.646578 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.647274 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:41.647466 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:41.647519 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:41.650302 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:34397: BlockingRecv error: recv got EOF from 127.21.28.67:34397 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 06:57:42 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:42 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:42.410138 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:42.405852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:49437)
0808 06:57:42.405992 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:42.406003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:42.406051 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:42.406168 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:42.406168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:42.406325 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:42.406376 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:42.406377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.406978 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.406979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:42.407509 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:42.407510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.407806 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.407806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:42.407849 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:42.408279 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:42.408284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:42.410081 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
W20260808 06:57:42.410250 2620 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:42.405931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:49437 (local address 127.21.28.126:40723)
0808 06:57:42.406045 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:42.406047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:42.406056 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:42.406094 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:42.406095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:42.406105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:42.406158 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:42.406430 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.406928 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.407559 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.407757 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.407800 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:42.410171 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:49437: BlockingRecv error: recv got EOF from 127.21.28.65:49437 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
Aug 08 06:57:42 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:42 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:42.522991 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:42.518570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:33541)
0808 06:57:42.518700 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:42.518710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:42.518773 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:42.518898 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:42.518899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:42.519053 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:42.519105 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:42.519107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.519892 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.519893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:42.520512 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:42.520515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.520822 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.520823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:42.520858 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:42.521261 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:42.521265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:42.522915 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
W20260808 06:57:42.523072 2620 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:42.518647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:33541 (local address 127.21.28.126:40723)
0808 06:57:42.518701 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:42.518703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:42.518768 (+ 65us) server_negotiation.cc:407] Connection header received
0808 06:57:42.518830 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:42.518830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:42.518840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:42.518878 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:42.519156 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.519798 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.520578 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.520769 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.520818 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:42.523019 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:33541: BlockingRecv error: recv got EOF from 127.21.28.66:33541 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 08 06:57:42 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:42 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:42.655618 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:42.650731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:60761)
0808 06:57:42.650870 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:42.650888 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:42.650947 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:42.651051 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:42.651052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:42.651209 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:42.651258 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:42.651260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.651903 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.651904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:42.652663 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:42.652666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.653110 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.653111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:42.653174 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:42.653657 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:42.653663 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:42.655539 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
W20260808 06:57:42.655723 2620 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:42.650791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:60761 (local address 127.21.28.126:40723)
0808 06:57:42.650845 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:42.650848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:42.650938 (+ 90us) server_negotiation.cc:407] Connection header received
0808 06:57:42.650987 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:42.650988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:42.651000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:42.651040 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:42.651300 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.651822 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.652746 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:42.653042 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:42.653107 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:42.655638 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:60761: BlockingRecv error: recv got EOF from 127.21.28.67:60761 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 08 06:57:43 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:43 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:43.414798 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:43.410628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:40055)
0808 06:57:43.410763 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:43.410773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:43.410823 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:43.410944 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:43.410945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:43.411105 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:43.411159 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:43.411160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.411805 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.411805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:43.412314 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:43.412316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.412604 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.412604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:43.412640 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:43.413042 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:43.413047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:43.414740 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
W20260808 06:57:43.414897 2625 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:43.410726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:40055 (local address 127.21.28.126:40723)
0808 06:57:43.410835 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:43.410837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:43.410845 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:43.410864 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:43.410865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:43.410875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:43.410931 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:43.411210 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.411754 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.412368 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.412559 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.412601 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:43.414826 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:40055: BlockingRecv error: recv got EOF from 127.21.28.65:40055 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
Aug 08 06:57:43 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:43 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:43.528075 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:43.523487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:57643)
0808 06:57:43.523615 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:43.523629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:43.523720 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:43.523852 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:43.523854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:43.524025 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:43.524074 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:43.524076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.524910 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.524911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:43.525575 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:43.525579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.525886 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.525887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:43.525924 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:43.526309 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:43.526312 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:43.527995 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
W20260808 06:57:43.528139 2625 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:43.523586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:57643 (local address 127.21.28.126:40723)
0808 06:57:43.523667 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:43.523693 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:43.523702 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:43.523783 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:43.523783 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:43.523795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:43.523840 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:43.524140 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.524833 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.525640 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.525830 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.525880 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:43.528087 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:57643: BlockingRecv error: recv got EOF from 127.21.28.66:57643 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 06:57:43 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:43 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:43.660789 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:43.656062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:58751)
0808 06:57:43.656191 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:43.656202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:43.656279 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:43.656438 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:43.656440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:43.656659 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:43.656719 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:43.656721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.657691 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.657692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:43.658286 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:43.658289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.658610 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.658611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:43.658650 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:43.659073 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:43.659078 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:43.660730 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
W20260808 06:57:43.660880 2625 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:43.656128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:58751 (local address 127.21.28.126:40723)
0808 06:57:43.656195 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:43.656197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:43.656257 (+ 60us) server_negotiation.cc:407] Connection header received
0808 06:57:43.656335 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:43.656336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:43.656349 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:43.656404 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:43.656819 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.657621 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.658351 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:43.658554 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:43.658607 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:43.660820 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:58751: BlockingRecv error: recv got EOF from 127.21.28.67:58751 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 06:57:44 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:44 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:44.419813 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:44.415254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:41629)
0808 06:57:44.415391 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:44.415403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:44.415461 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:44.415590 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:44.415590 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:44.415774 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:44.415826 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:44.415827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.416454 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.416454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:44.416998 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:44.417002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.417466 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.417467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:44.417514 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:44.417958 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:44.417962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:44.419753 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
W20260808 06:57:44.419929 2634 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:44.415310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:41629 (local address 127.21.28.126:40723)
0808 06:57:44.415423 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:44.415425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:44.415446 (+ 21us) server_negotiation.cc:407] Connection header received
0808 06:57:44.415506 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:44.415507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:44.415518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:44.415579 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:44.415879 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.416388 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.417084 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.417401 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.417452 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:44.419847 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:41629: BlockingRecv error: recv got EOF from 127.21.28.65:41629 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
Aug 08 06:57:44 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:44 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:44.532622 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:44.528496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:58821)
0808 06:57:44.528621 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:44.528631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:44.528691 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:44.528804 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:44.528805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:44.528976 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:44.529033 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:44.529035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.529641 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.529642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:44.530153 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:44.530154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.530435 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.530436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:44.530472 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:44.530869 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:44.530873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:44.532562 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
W20260808 06:57:44.532732 2634 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:44.528547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:58821 (local address 127.21.28.126:40723)
0808 06:57:44.528597 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:44.528598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:44.528686 (+ 88us) server_negotiation.cc:407] Connection header received
0808 06:57:44.528733 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:44.528734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:44.528745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:44.528793 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:44.529085 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.529587 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.530206 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.530384 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.530435 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:44.532670 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:58821: BlockingRecv error: recv got EOF from 127.21.28.66:58821 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 08 06:57:44 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:44 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:44.665400 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:44.661265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:43341)
0808 06:57:44.661392 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:44.661404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:44.661467 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:44.661567 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:44.661567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:44.661723 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:44.661772 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:44.661773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.662356 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.662356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:44.662884 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:44.662886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.663153 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.663153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:44.663190 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:44.663576 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:44.663580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:44.665338 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
W20260808 06:57:44.665504 2634 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:44.661330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:43341 (local address 127.21.28.126:40723)
0808 06:57:44.661377 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:44.661379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:44.661452 (+ 73us) server_negotiation.cc:407] Connection header received
0808 06:57:44.661506 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:44.661507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:44.661517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:44.661561 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:44.661818 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.662307 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.662937 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:44.663108 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:44.663148 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:44.665441 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:43341: BlockingRecv error: recv got EOF from 127.21.28.67:43341 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 08 06:57:45 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:45 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:45.424535 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:45.420198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:44583)
0808 06:57:45.420330 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:45.420343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:45.420411 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:45.420535 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:45.420535 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:45.420698 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:45.420750 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:45.420751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.421539 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.421540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:45.422061 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:45.422062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.422371 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.422371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:45.422413 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:45.422812 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:45.422816 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:45.424476 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
W20260808 06:57:45.424630 2640 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:45.420251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:44583 (local address 127.21.28.126:40723)
0808 06:57:45.420381 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:45.420383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:45.420391 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:45.420460 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:45.420461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:45.420470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:45.420524 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:45.420819 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.421471 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.422127 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.422323 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.422367 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:45.424564 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:44583: BlockingRecv error: recv got EOF from 127.21.28.65:44583 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
Aug 08 06:57:45 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:45 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:45.537158 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:45.533108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:48585)
0808 06:57:45.533249 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:45.533258 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:45.533320 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:45.533435 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:45.533435 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:45.533600 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:45.533651 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:45.533653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.534240 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.534241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:45.534762 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:45.534765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.535039 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.535039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:45.535077 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:45.535477 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:45.535482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:45.537095 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
W20260808 06:57:45.537272 2640 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:45.533162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:48585 (local address 127.21.28.126:40723)
0808 06:57:45.533239 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:45.533241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:45.533315 (+ 74us) server_negotiation.cc:407] Connection header received
0808 06:57:45.533361 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:45.533362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:45.533372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:45.533430 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:45.533699 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.534185 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.534818 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.534991 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.535031 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:45.537205 (+ 2174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:48585: BlockingRecv error: recv got EOF from 127.21.28.66:48585 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 06:57:45 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:45 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:45.670215 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:45.665807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:50513)
0808 06:57:45.665993 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:45.666003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:45.666068 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:45.666204 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:45.666205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:45.666374 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:45.666423 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:45.666425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.667049 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.667050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:45.667582 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:45.667584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.667938 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.667939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:45.667994 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:45.668418 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:45.668422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:45.670145 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":62,"threads_started":1}
W20260808 06:57:45.670293 2640 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:45.665924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:50513 (local address 127.21.28.126:40723)
0808 06:57:45.665972 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:45.665974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:45.666057 (+ 83us) server_negotiation.cc:407] Connection header received
0808 06:57:45.666125 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:45.666126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:45.666139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:45.666181 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:45.666488 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.666978 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.667655 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:45.667874 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:45.667929 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:45.670230 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:50513: BlockingRecv error: recv got EOF from 127.21.28.67:50513 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 06:57:46 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:46 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:46.432091 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:46.424970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:41413)
0808 06:57:46.425116 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:46.425125 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:46.425169 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:46.425352 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:46.425353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:46.425547 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:46.425597 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:46.425599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.426252 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.426253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:46.426845 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:46.426848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.427186 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.427187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:46.427240 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:46.430249 (+ 3009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:46.430254 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:46.432016 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
W20260808 06:57:46.432173 2649 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:46.425034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:41413 (local address 127.21.28.126:40723)
0808 06:57:46.425144 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:46.425146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:46.425156 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:46.425226 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:46.425226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:46.425238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:46.425343 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:46.425674 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.426180 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.426930 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.427125 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.427181 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:46.432108 (+ 4927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:41413: BlockingRecv error: recv got EOF from 127.21.28.65:41413 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
Aug 08 06:57:46 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:46 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:46.541793 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:46.537660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:36623)
0808 06:57:46.537808 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:46.537817 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:46.537878 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:46.537996 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:46.537997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:46.538164 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:46.538216 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:46.538218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.538818 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.538818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:46.539357 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:46.539358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.539637 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.539638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:46.539699 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:46.540119 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:46.540124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:46.541702 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
W20260808 06:57:46.541862 2649 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:46.537724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:36623 (local address 127.21.28.126:40723)
0808 06:57:46.537780 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:46.537782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:46.537871 (+ 89us) server_negotiation.cc:407] Connection header received
0808 06:57:46.537932 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:46.537933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:46.537944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:46.537981 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:46.538263 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.538766 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.539410 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.539591 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.539635 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:46.541808 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:36623: BlockingRecv error: recv got EOF from 127.21.28.66:36623 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 08 06:57:46 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:46 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:46.674803 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:46.670666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:47021)
0808 06:57:46.670816 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:46.670827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:46.670897 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:46.671011 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:46.671012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:46.671176 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:46.671226 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:46.671227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.671856 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.671857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:46.672381 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:46.672382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.672654 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.672655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:46.672691 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:46.673085 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:46.673089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:46.674725 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
W20260808 06:57:46.674881 2649 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:46.670716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:47021 (local address 127.21.28.126:40723)
0808 06:57:46.670770 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:46.670773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:46.670877 (+ 104us) server_negotiation.cc:407] Connection header received
0808 06:57:46.670941 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:46.670942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:46.670955 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:46.671007 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:46.671269 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.671795 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.672436 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:46.672609 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:46.672650 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:46.674817 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:47021: BlockingRecv error: recv got EOF from 127.21.28.67:47021 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 08 06:57:47 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:47 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:47.436725 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:47.432505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:51483)
0808 06:57:47.432632 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:47.432644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:47.432708 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:47.432820 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:47.432820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:47.432993 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:47.433046 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:47.433048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.433650 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.433650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:47.434191 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:47.434193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.434515 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.434515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:47.434550 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:47.434963 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:47.434968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:47.436667 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
W20260808 06:57:47.436825 2655 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:47.432570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:51483 (local address 127.21.28.126:40723)
0808 06:57:47.432707 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:47.432710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:47.432718 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:47.432742 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:47.432742 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:47.432753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:47.432809 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:47.433098 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.433598 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.434256 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.434464 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.434512 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:47.436757 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:51483: BlockingRecv error: recv got EOF from 127.21.28.65:51483 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
Aug 08 06:57:47 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:47 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:47.546734 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:47.542371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:55317)
0808 06:57:47.542520 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:47.542531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:47.542594 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:47.542709 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:47.542710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:47.542889 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:47.542945 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:47.542947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.543555 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.543556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:47.544112 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:47.544114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.544388 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.544389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:47.544426 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:47.544845 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:47.544850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:47.546674 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
W20260808 06:57:47.546820 2655 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:47.542429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:55317 (local address 127.21.28.126:40723)
0808 06:57:47.542499 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:47.542501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:47.542590 (+ 89us) server_negotiation.cc:407] Connection header received
0808 06:57:47.542649 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:47.542649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:47.542660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:47.542698 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:47.543002 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.543500 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.544163 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.544343 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.544400 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:47.546762 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:55317: BlockingRecv error: recv got EOF from 127.21.28.66:55317 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 06:57:47 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:47 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:47.680873 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:47.675197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:38743)
0808 06:57:47.675326 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:47.675338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:47.675405 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:47.675559 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:47.675560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:47.675797 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:47.675856 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:47.675859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.676801 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.676802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:47.677661 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:47.677664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.678167 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.678167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:47.678222 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:47.678728 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:47.678734 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:47.680783 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
W20260808 06:57:47.681124 2655 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:47.675267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:38743 (local address 127.21.28.126:40723)
0808 06:57:47.675331 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:47.675333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:47.675384 (+ 51us) server_negotiation.cc:407] Connection header received
0808 06:57:47.675464 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:47.675465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:47.675482 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:47.675537 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:47.675946 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.676724 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.677739 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:47.678097 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:47.678167 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:47.681062 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:38743: BlockingRecv error: recv got EOF from 127.21.28.67:38743 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 08 06:57:48 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:48 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:48.441813 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:48.437261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:58451)
0808 06:57:48.437422 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:48.437432 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:48.437484 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:48.437616 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:48.437616 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:48.437784 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:48.437838 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:48.437840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.438536 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.438538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:48.439145 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:48.439148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.439502 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.439502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:48.439541 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:48.439993 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:48.439999 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:48.441754 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
W20260808 06:57:48.441951 2662 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:48.437349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:58451 (local address 127.21.28.126:40723)
0808 06:57:48.437470 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:48.437471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:48.437481 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:57:48.437540 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:48.437540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:48.437553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:48.437606 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:48.437889 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.438472 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.439221 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.439438 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.439493 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:48.441868 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:58451: BlockingRecv error: recv got EOF from 127.21.28.65:58451 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
Aug 08 06:57:48 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:48 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:48.551515 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:48.547158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:48307)
0808 06:57:48.547303 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:48.547315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:48.547389 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:48.547529 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:48.547530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:48.547733 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:48.547783 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:48.547785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.548444 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.548445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:48.548967 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:48.548969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.549290 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.549291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:48.549347 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:48.549770 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:48.549774 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:48.551436 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
W20260808 06:57:48.551589 2662 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:48.547262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:48307 (local address 127.21.28.126:40723)
0808 06:57:48.547326 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:48.547328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:48.547370 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:57:48.547453 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:48.547454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:48.547466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:48.547516 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:48.547854 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.548359 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.549042 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.549230 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.549285 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:48.551535 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:48307: BlockingRecv error: recv got EOF from 127.21.28.66:48307 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 08 06:57:48 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:48 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:48.685957 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:48.681373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:52455)
0808 06:57:48.681506 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:48.681516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:48.681573 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:48.681705 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:48.681705 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:48.681898 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:48.681948 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:48.681951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.682694 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.682695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:48.683261 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:48.683263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.683622 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.683623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:48.683660 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:48.684154 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:48.684159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:48.685885 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
W20260808 06:57:48.686034 2662 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:48.681450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:52455 (local address 127.21.28.126:40723)
0808 06:57:48.681512 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:48.681514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:48.681567 (+ 53us) server_negotiation.cc:407] Connection header received
0808 06:57:48.681624 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:48.681625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:48.681636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:48.681692 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:48.682005 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.682594 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.683358 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:48.683555 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:48.683614 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:48.685975 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:52455: BlockingRecv error: recv got EOF from 127.21.28.67:52455 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 08 06:57:49 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:49 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:49.446647 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:49.442267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:35767)
0808 06:57:49.442415 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:49.442427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:49.442486 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:49.442596 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:49.442597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:49.442761 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:49.442813 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:49.442815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.443405 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.443405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:49.443958 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:49.443960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.444239 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.444240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:49.444277 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:49.444802 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:49.444807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:49.446577 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
W20260808 06:57:49.446817 2668 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:49.442339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:35767 (local address 127.21.28.126:40723)
0808 06:57:49.442477 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:49.442479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:49.442487 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:49.442525 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:49.442526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:49.442536 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:49.442591 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:49.442861 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.443345 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.444006 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.444189 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.444234 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:49.446736 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:35767: BlockingRecv error: recv got EOF from 127.21.28.65:35767 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
Aug 08 06:57:49 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:49 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:49.556399 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:49.551971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:44957)
0808 06:57:49.552108 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:49.552120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:49.552189 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:49.552311 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:49.552312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:49.552494 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:49.552541 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:49.552543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.553187 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.553188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:49.553726 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:49.553728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.554066 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.554067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:49.554123 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:49.554526 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:49.554531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:49.556306 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
W20260808 06:57:49.556465 2668 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:49.552038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:44957 (local address 127.21.28.126:40723)
0808 06:57:49.552108 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:49.552110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:49.552180 (+ 70us) server_negotiation.cc:407] Connection header received
0808 06:57:49.552242 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:49.552244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:49.552255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:49.552302 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:49.552614 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.553110 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.553802 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.554005 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.554059 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:49.556401 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:44957: BlockingRecv error: recv got EOF from 127.21.28.66:44957 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 06:57:49 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:49 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.67@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:49.690680 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:49.686410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.67:40993)
0808 06:57:49.686536 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:49.686547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:49.686608 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:49.686736 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:49.686737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:49.686912 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:49.686961 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:49.686963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.687575 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.687576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:49.688139 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:49.688142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.688458 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.688459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:49.688497 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:49.688952 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:49.688956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:49.690620 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
W20260808 06:57:49.690790 2668 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:49.686461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:40993 (local address 127.21.28.126:40723)
0808 06:57:49.686535 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:49.686537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:49.686592 (+ 55us) server_negotiation.cc:407] Connection header received
0808 06:57:49.686662 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:49.686663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:49.686677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:49.686719 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:49.687012 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.687514 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.688208 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:49.688396 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:49.688450 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:49.690721 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.67:40993: BlockingRecv error: recv got EOF from 127.21.28.67:40993 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 06:57:50 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:50 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.65@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:50.451399 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:50.447106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.65:51889)
0808 06:57:50.447242 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:50.447253 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:50.447309 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:50.447428 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:50.447428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:50.447589 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:50.447641 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:50.447643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.448303 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.448303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:50.448820 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:50.448822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.449175 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.449176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:50.449232 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:50.449692 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:50.449698 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:50.451332 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
W20260808 06:57:50.451489 2676 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:50.447189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:51889 (local address 127.21.28.126:40723)
0808 06:57:50.447298 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:50.447300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:50.447309 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:57:50.447351 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:50.447352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:50.447362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:50.447419 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:50.447737 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.448244 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.448873 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.449084 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.449141 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:50.451423 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.65:51889: BlockingRecv error: recv got EOF from 127.21.28.65:51889 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
Aug 08 06:57:50 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:57:50 dist-test-slave-nj21 krb5kdc[1061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.28.66@KRBTEST.COM for oryx/127.21.28.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:57:50.561450 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:50.556852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:40723 (local address 127.21.28.66:49795)
0808 06:57:50.556980 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:50.556992 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:50.557070 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:50.557204 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:50.557204 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:50.557378 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:50.557440 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:50.557442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.558090 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.558091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:50.558616 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:50.558617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.559005 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.559007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:50.559060 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:50.559502 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:50.559508 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:50.561388 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.28.126:40723: Server oryx/127.21.28.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
W20260808 06:57:50.561542 2676 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:50.556927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:49795 (local address 127.21.28.126:40723)
0808 06:57:50.557003 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:50.557005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:50.557061 (+ 56us) server_negotiation.cc:407] Connection header received
0808 06:57:50.557131 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:50.557132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:50.557143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:50.557192 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:50.557514 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.558017 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.558694 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.558947 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.559004 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:50.561479 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.28.66:49795: BlockingRecv error: recv got EOF from 127.21.28.66:49795 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:57:50.611990 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 1936
I20260808 06:57:50.624828 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 2182
I20260808 06:57:50.635592 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 2376
I20260808 06:57:50.644451 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 1835
2026-08-08T06:57:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17522 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T06:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:50Z Disabled control of system clock
I20260808 06:57:50.675016 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:45999
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:45999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 06:57:50.754024 2687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:50.754209 2687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:50.754228 2687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:50.755313 2687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:50.755347 2687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:50.755358 2687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:50.755369 2687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:50.755383 2687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:50.756690 2687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:45999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:50.756922 2687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:50.757191 2687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:50.762243 2695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:50.762246 2693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:50.762269 2692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:50.762403 2687 server_base.cc:1034] running on GCE node
I20260808 06:57:50.762657 2687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:50.762936 2687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:50.764072 2687 hybrid_clock.cc:648] HybridClock initialized: now 1786172270764057 us; error 28 us; skew 500 ppm
I20260808 06:57:50.765213 2687 webserver.cc:460] Webserver started at http://127.21.28.126:39951/ using document root <none> and password file <none>
I20260808 06:57:50.765496 2687 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:50.765559 2687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:50.765684 2687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:50.766682 2687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "cf5e1fd14a1c49a0a5ddd77515476d23"
format_stamp: "Formatted at 2026-08-08 06:57:50 on dist-test-slave-nj21"
server_key: "b76a840b9f452d5593e52c7178d27339"
server_key_iv: "4d817742ebdfb51ad866ba7a653abb81"
server_key_version: "encryptionkey@0"
I20260808 06:57:50.767002 2687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "cf5e1fd14a1c49a0a5ddd77515476d23"
format_stamp: "Formatted at 2026-08-08 06:57:50 on dist-test-slave-nj21"
server_key: "b76a840b9f452d5593e52c7178d27339"
server_key_iv: "4d817742ebdfb51ad866ba7a653abb81"
server_key_version: "encryptionkey@0"
I20260808 06:57:50.768275 2687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:50.768996 2706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:50.769137 2687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:50.769230 2687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "cf5e1fd14a1c49a0a5ddd77515476d23"
format_stamp: "Formatted at 2026-08-08 06:57:50 on dist-test-slave-nj21"
server_key: "b76a840b9f452d5593e52c7178d27339"
server_key_iv: "4d817742ebdfb51ad866ba7a653abb81"
server_key_version: "encryptionkey@0"
I20260808 06:57:50.769291 2687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:50.790375 2687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:50.790622 2687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:50.790727 2687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:50.794370 2687 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:45999
I20260808 06:57:50.794415 2794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:45999 every 8 connection(s)
I20260808 06:57:50.794708 2687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:50.795373 2795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:50.797931 2795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23: Bootstrap starting.
I20260808 06:57:50.798501 2795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:50.798785 2795 log.cc:824] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:50.799103 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 2687
I20260808 06:57:50.799201 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:50.799368 21617 external_mini_cluster.cc:1428] Setting key 9d40ae21b56f077fb9cf065b52f85913
I20260808 06:57:50.799446 2795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23: No bootstrap required, opened a new log
I20260808 06:57:50.801671 2795 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5e1fd14a1c49a0a5ddd77515476d23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45999 } }
I20260808 06:57:50.801836 2795 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:50.801872 2795 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5e1fd14a1c49a0a5ddd77515476d23, State: Initialized, Role: FOLLOWER
I20260808 06:57:50.801997 2795 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5e1fd14a1c49a0a5ddd77515476d23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45999 } }
I20260808 06:57:50.802094 2795 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:50.802136 2795 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:50.802186 2795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:50.802867 2795 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5e1fd14a1c49a0a5ddd77515476d23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45999 } }
I20260808 06:57:50.802969 2795 leader_election.cc:302] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5e1fd14a1c49a0a5ddd77515476d23; no voters:
I20260808 06:57:50.803108 2795 leader_election.cc:288] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:50.803145 2801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:50.803331 2795 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:50.803339 2801 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [term 1 LEADER]: Becoming Leader. State: Replica: cf5e1fd14a1c49a0a5ddd77515476d23, State: Running, Role: LEADER
I20260808 06:57:50.803545 2801 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5e1fd14a1c49a0a5ddd77515476d23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45999 } }
I20260808 06:57:50.803961 2806 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf5e1fd14a1c49a0a5ddd77515476d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5e1fd14a1c49a0a5ddd77515476d23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45999 } } }
I20260808 06:57:50.804033 2806 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:50.804150 2806 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5e1fd14a1c49a0a5ddd77515476d23. Latest consensus state: current_term: 1 leader_uuid: "cf5e1fd14a1c49a0a5ddd77515476d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5e1fd14a1c49a0a5ddd77515476d23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45999 } } }
I20260808 06:57:50.804188 2806 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23 [sys.catalog]: This master's current role is: LEADER
W20260808 06:57:50.805065 2825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:57:50.806260 2825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:57:50.806368 2818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:50.806541 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:50.799849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.21.28.126:45999)
0808 06:57:50.800132 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:50.800142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:50.800176 (+ 34us) server_negotiation.cc:407] Connection header received
0808 06:57:50.800772 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:50.800781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:50.800856 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:50.801026 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:50.801178 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.803219 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.804282 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.804886 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.805075 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:50.805694 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:50.805698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:50.805699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:50.805880 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:50.805933 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:50.805947 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:50.806013 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:50.806199 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 06:57:50.806284 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":141,"threads_started":1}
I20260808 06:57:50.806579 2818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:50.808063 2818 catalog_manager.cc:1347] Generated new cluster ID: cb1785df92fd4586b2f5b2b58be8c0b3
I20260808 06:57:50.808116 2818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:50.824112 2818 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:50.824599 2818 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:50.826516 2818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf5e1fd14a1c49a0a5ddd77515476d23: Generated new TSK 0
I20260808 06:57:50.826675 2818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:57:50.835619 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45999
--builtin_ntp_servers=127.21.28.84:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 06:57:50.915337 2829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:50.915531 2829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:50.915551 2829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:50.916685 2829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:50.916724 2829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:50.916757 2829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:50.918035 2829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:50.918239 2829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:50.918509 2829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:50.923975 2841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:50.923987 2842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:50.923988 2845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:50.924170 2829 server_base.cc:1034] running on GCE node
I20260808 06:57:50.924363 2829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:50.924579 2829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:50.925751 2829 hybrid_clock.cc:648] HybridClock initialized: now 1786172270925730 us; error 32 us; skew 500 ppm
I20260808 06:57:50.926954 2829 webserver.cc:460] Webserver started at http://127.21.28.65:34353/ using document root <none> and password file <none>
I20260808 06:57:50.927156 2829 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:50.927203 2829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:50.927348 2829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:50.928382 2829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "4076cbae121b4e3d8a2217ebb52d0fb6"
format_stamp: "Formatted at 2026-08-08 06:57:50 on dist-test-slave-nj21"
server_key: "70e4a08f4f57d09a3aabcb382c517a93"
server_key_iv: "3ab7725e15ff0b7108f06f83c9c4a50e"
server_key_version: "encryptionkey@0"
I20260808 06:57:50.928694 2829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "4076cbae121b4e3d8a2217ebb52d0fb6"
format_stamp: "Formatted at 2026-08-08 06:57:50 on dist-test-slave-nj21"
server_key: "70e4a08f4f57d09a3aabcb382c517a93"
server_key_iv: "3ab7725e15ff0b7108f06f83c9c4a50e"
server_key_version: "encryptionkey@0"
I20260808 06:57:50.930042 2829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 06:57:50.930732 2858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:50.930898 2829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:50.930979 2829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "4076cbae121b4e3d8a2217ebb52d0fb6"
format_stamp: "Formatted at 2026-08-08 06:57:50 on dist-test-slave-nj21"
server_key: "70e4a08f4f57d09a3aabcb382c517a93"
server_key_iv: "3ab7725e15ff0b7108f06f83c9c4a50e"
server_key_version: "encryptionkey@0"
I20260808 06:57:50.931041 2829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:50.941604 2829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:50.941849 2829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:50.941972 2829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:50.942157 2829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:50.942452 2829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:50.942485 2829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:50.942512 2829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:50.942528 2829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:50.947926 2829 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:46397
I20260808 06:57:50.947992 3023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:46397 every 8 connection(s)
I20260808 06:57:50.948251 2829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:50.950510 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 2829
I20260808 06:57:50.950600 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:50.950766 21617 external_mini_cluster.cc:1428] Setting key 5ace8aa5657dfab01081e112067b50b9
I20260808 06:57:50.951608 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45999
--builtin_ntp_servers=127.21.28.84:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 06:57:50.953435 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:50.948892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:51933 (local address 127.21.28.126:45999)
0808 06:57:50.948935 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:50.948937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:50.949134 (+ 197us) server_negotiation.cc:407] Connection header received
0808 06:57:50.949651 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:50.949653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:50.949673 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:50.949729 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:50.950447 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.951045 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.952104 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.952313 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.952369 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:50.953213 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:50.953215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:50.953215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:50.953219 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:50.953236 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:50.953240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:50.953285 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:50.953357 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 06:57:50.953378 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 06:57:50.953672 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:50.948865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45999 (local address 127.21.28.65:51933)
0808 06:57:50.949024 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:50.949054 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:50.949553 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:50.949824 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:50.949827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:50.949834 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:50.950337 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:50.950342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.951121 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.951122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:50.952028 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:50.952031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:50.952394 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:50.952396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:50.952513 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:50.953061 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:50.953061 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:50.953098 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:50.953124 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:50.953296 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:50.953300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:50.953303 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:50.953362 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 06:57:50.953465 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260808 06:57:50.954213 3024 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45999
I20260808 06:57:50.954313 3024 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:50.954487 3024 heartbeater.cc:511] Master 127.21.28.126:45999 requested a full tablet report, sending...
I20260808 06:57:50.954991 2737 ts_manager.cc:194] Registered new tserver with Master: 4076cbae121b4e3d8a2217ebb52d0fb6 (127.21.28.65:46397)
I20260808 06:57:50.955593 2737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.65:51933
W20260808 06:57:51.032469 3030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:51.032655 3030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:51.032675 3030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:51.033813 3030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:51.033851 3030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:51.033876 3030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:51.035185 3030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:51.035383 3030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:51.035635 3030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:57:51.040441 3030 server_base.cc:1034] running on GCE node
W20260808 06:57:51.040408 3047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.040429 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.040396 3041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:51.040802 3030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:51.041043 3030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:51.042181 3030 hybrid_clock.cc:648] HybridClock initialized: now 1786172271042161 us; error 34 us; skew 500 ppm
I20260808 06:57:51.043354 3030 webserver.cc:460] Webserver started at http://127.21.28.66:34171/ using document root <none> and password file <none>
I20260808 06:57:51.043565 3030 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:51.043612 3030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:51.043766 3030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:51.044790 3030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "5fa424a44dfd4e218057a3fd57577a26"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "097128d87fc666aa9160bb959244980b"
server_key_iv: "59d16bf87ddad82e2cb3c1886b01b5fe"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.045131 3030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "5fa424a44dfd4e218057a3fd57577a26"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "097128d87fc666aa9160bb959244980b"
server_key_iv: "59d16bf87ddad82e2cb3c1886b01b5fe"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.046317 3030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:51.046954 3055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.047118 3030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:51.047226 3030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "5fa424a44dfd4e218057a3fd57577a26"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "097128d87fc666aa9160bb959244980b"
server_key_iv: "59d16bf87ddad82e2cb3c1886b01b5fe"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.047298 3030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:51.058277 3030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:51.058508 3030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:51.058622 3030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:51.058815 3030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:51.059109 3030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:51.059142 3030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.059170 3030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:51.059193 3030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.064747 3030 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:34185
I20260808 06:57:51.064803 3215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:34185 every 8 connection(s)
I20260808 06:57:51.065068 3030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:51.066972 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 3030
I20260808 06:57:51.067044 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:51.067226 21617 external_mini_cluster.cc:1428] Setting key 235b02f255ec4c80bb4a91bfb86eb221
I20260808 06:57:51.068145 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45999
--builtin_ntp_servers=127.21.28.84:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 06:57:51.070158 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.065653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:54049 (local address 127.21.28.126:45999)
0808 06:57:51.065699 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:51.065701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:51.065928 (+ 227us) server_negotiation.cc:407] Connection header received
0808 06:57:51.066417 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:51.066418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:51.066427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:51.066469 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:51.067076 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:51.067780 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:51.068827 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:51.069081 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:51.069150 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:51.069937 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:51.069938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:51.069938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:51.069942 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:51.069956 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:51.069960 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:51.070000 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:51.070079 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 06:57:51.070100 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:51.070399 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.065629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45999 (local address 127.21.28.66:54049)
0808 06:57:51.065814 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:51.065857 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:51.066318 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:51.066527 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:51.066529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:51.066537 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:51.066994 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:51.066998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:51.067870 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:51.067871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:51.068750 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:51.068752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:51.069153 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:51.069154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:51.069265 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:51.069793 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:51.069794 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:51.069830 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:51.069857 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:51.070011 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:51.070016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:51.070019 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:51.070079 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 06:57:51.070179 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260808 06:57:51.070891 3217 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45999
I20260808 06:57:51.070981 3217 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:51.071146 3217 heartbeater.cc:511] Master 127.21.28.126:45999 requested a full tablet report, sending...
I20260808 06:57:51.071483 2737 ts_manager.cc:194] Registered new tserver with Master: 5fa424a44dfd4e218057a3fd57577a26 (127.21.28.66:34185)
I20260808 06:57:51.072019 2737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.66:54049
W20260808 06:57:51.148011 3220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:51.148196 3220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:51.148223 3220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:51.149349 3220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:51.149389 3220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:51.149411 3220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:51.150770 3220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:51.150949 3220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:51.151183 3220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:51.155890 3236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.155925 3233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:51.155958 3220 server_base.cc:1034] running on GCE node
W20260808 06:57:51.155958 3230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:51.156311 3220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:51.156530 3220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:51.157663 3220 hybrid_clock.cc:648] HybridClock initialized: now 1786172271157649 us; error 31 us; skew 500 ppm
I20260808 06:57:51.158838 3220 webserver.cc:460] Webserver started at http://127.21.28.67:33125/ using document root <none> and password file <none>
I20260808 06:57:51.159060 3220 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:51.159104 3220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:51.159216 3220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:51.160213 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "a6b6a64efe4a4773a6d9cd42ca5ab24c"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "354924a56240eba35d46a2e3882b35fe"
server_key_iv: "7a334cab6aa4252cb8147746601f33a2"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.160519 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "a6b6a64efe4a4773a6d9cd42ca5ab24c"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "354924a56240eba35d46a2e3882b35fe"
server_key_iv: "7a334cab6aa4252cb8147746601f33a2"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.161677 3220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:51.162304 3245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.162452 3220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:51.162516 3220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "a6b6a64efe4a4773a6d9cd42ca5ab24c"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "354924a56240eba35d46a2e3882b35fe"
server_key_iv: "7a334cab6aa4252cb8147746601f33a2"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.162580 3220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:51.173964 3220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:51.174201 3220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:51.174306 3220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:51.174528 3220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:51.174832 3220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:51.174866 3220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.174885 3220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:51.174907 3220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.180541 3220 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:43279
I20260808 06:57:51.180579 3433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:43279 every 8 connection(s)
I20260808 06:57:51.180859 3220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:51.183314 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 3220
I20260808 06:57:51.183422 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:51.183645 21617 external_mini_cluster.cc:1428] Setting key 1f630e8f486ac189776c88c9a2011fd4
I20260808 06:57:51.185738 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.181490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:39687 (local address 127.21.28.126:45999)
0808 06:57:51.181545 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:51.181548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:51.181742 (+ 194us) server_negotiation.cc:407] Connection header received
0808 06:57:51.182209 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:51.182210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:51.182220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:51.182261 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:51.182842 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:51.183323 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:51.184637 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:51.184832 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:51.184883 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:51.185550 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:51.185551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:51.185551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:51.185555 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:51.185569 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:51.185572 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:51.185609 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:51.185661 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 06:57:51.185681 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 06:57:51.185932 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.181452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45999 (local address 127.21.28.67:39687)
0808 06:57:51.181630 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:51.181667 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:51.182118 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:51.182310 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:51.182313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:51.182321 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:51.182783 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:51.182787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:51.183410 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:51.183411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:51.184569 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:51.184573 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:51.184901 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:51.184902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:51.185004 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:51.185435 (+ 431us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:51.185436 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:51.185461 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:51.185483 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:51.185614 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:51.185617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:51.185619 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:51.185662 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 06:57:51.185734 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260808 06:57:51.186345 3435 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45999
I20260808 06:57:51.186430 3435 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:51.186592 3435 heartbeater.cc:511] Master 127.21.28.126:45999 requested a full tablet report, sending...
I20260808 06:57:51.186935 2737 ts_manager.cc:194] Registered new tserver with Master: a6b6a64efe4a4773a6d9cd42ca5ab24c (127.21.28.67:43279)
I20260808 06:57:51.187288 2737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.67:39687
I20260808 06:57:51.195163 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 06:57:51.196924 2800 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:51.196447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.21.28.126:45999)
0808 06:57:51.196504 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:51.196506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:51.196606 (+ 100us) server_negotiation.cc:407] Connection header received
0808 06:57:51.196666 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:51.196667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:51.196676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:51.196716 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:51.196860 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43902: BlockingRecv error: recv got EOF from 127.0.0.1:43902 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:57:51.196982 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 2829
I20260808 06:57:51.210907 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 3030
I20260808 06:57:51.223769 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 3220
I20260808 06:57:51.231483 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 2687
2026-08-08T06:57:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (585 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T06:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:51Z Disabled control of system clock
I20260808 06:57:51.260293 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:44681
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:44681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 06:57:51.338968 3454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:51.339157 3454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:51.339177 3454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:51.340318 3454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:51.340361 3454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:51.340373 3454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:51.340384 3454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:51.340394 3454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:51.341657 3454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:44681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:44681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:51.341843 3454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:51.342069 3454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:51.346524 3466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.346572 3461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:51.346673 3454 server_base.cc:1034] running on GCE node
W20260808 06:57:51.346733 3463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:51.346972 3454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:51.347254 3454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:51.348392 3454 hybrid_clock.cc:648] HybridClock initialized: now 1786172271348373 us; error 36 us; skew 500 ppm
I20260808 06:57:51.349514 3454 webserver.cc:460] Webserver started at http://127.21.28.126:37899/ using document root <none> and password file <none>
I20260808 06:57:51.349726 3454 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:51.349773 3454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:51.349908 3454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:51.350945 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "f11607f00fcc42409b7523438c348102"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "d444a77433c60065c71d87c74b188e36"
server_key_iv: "7aaf44f711d0f037b315ebb64943d089"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.351261 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "f11607f00fcc42409b7523438c348102"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "d444a77433c60065c71d87c74b188e36"
server_key_iv: "7aaf44f711d0f037b315ebb64943d089"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.352479 3454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:51.353117 3474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.353266 3454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:51.353348 3454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "f11607f00fcc42409b7523438c348102"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "d444a77433c60065c71d87c74b188e36"
server_key_iv: "7aaf44f711d0f037b315ebb64943d089"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.353408 3454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:51.356602 3454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:51.356829 3454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:51.356952 3454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:51.360512 3454 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:44681
I20260808 06:57:51.360546 3562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:44681 every 8 connection(s)
I20260808 06:57:51.360816 3454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:51.361411 3563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:51.363770 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 3454
I20260808 06:57:51.363893 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:51.364028 3563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102: Bootstrap starting.
I20260808 06:57:51.364090 21617 external_mini_cluster.cc:1428] Setting key fe6e8d5e19ec2a4fed37aded6132a41c
I20260808 06:57:51.364686 3563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:51.364971 3563 log.cc:824] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:51.365594 3563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102: No bootstrap required, opened a new log
I20260808 06:57:51.366415 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.364564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.21.28.126:44681)
0808 06:57:51.364751 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:51.364760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:51.364790 (+ 30us) server_negotiation.cc:407] Connection header received
0808 06:57:51.365204 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:51.365210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:51.365246 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:51.365371 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:51.366042 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:51.366046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:51.366046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:51.366051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:51.366090 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:51.366099 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:51.366173 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:51.366206 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 06:57:51.366259 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260808 06:57:51.367195 3563 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11607f00fcc42409b7523438c348102" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44681 } }
I20260808 06:57:51.367324 3563 raft_consensus.cc:374] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:51.367347 3563 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11607f00fcc42409b7523438c348102, State: Initialized, Role: FOLLOWER
I20260808 06:57:51.367427 3563 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11607f00fcc42409b7523438c348102" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44681 } }
I20260808 06:57:51.367473 3563 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:51.367497 3563 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:51.367527 3563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:51.368301 3563 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11607f00fcc42409b7523438c348102" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44681 } }
I20260808 06:57:51.368398 3563 leader_election.cc:302] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11607f00fcc42409b7523438c348102; no voters:
I20260808 06:57:51.368517 3563 leader_election.cc:288] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:51.368564 3573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:51.368710 3573 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [term 1 LEADER]: Becoming Leader. State: Replica: f11607f00fcc42409b7523438c348102, State: Running, Role: LEADER
I20260808 06:57:51.368758 3563 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:51.368795 3573 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11607f00fcc42409b7523438c348102" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44681 } }
I20260808 06:57:51.369148 3574 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f11607f00fcc42409b7523438c348102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11607f00fcc42409b7523438c348102" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44681 } } }
I20260808 06:57:51.369251 3574 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:51.369156 3576 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11607f00fcc42409b7523438c348102. Latest consensus state: current_term: 1 leader_uuid: "f11607f00fcc42409b7523438c348102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11607f00fcc42409b7523438c348102" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44681 } } }
I20260808 06:57:51.369625 3576 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:51.369979 3583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:51.370432 3583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:51.371870 3583 catalog_manager.cc:1347] Generated new cluster ID: e50a8025ad9e40afb4eab0bf97b0cb7c
I20260808 06:57:51.371920 3583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:51.401062 3583 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:51.401248 3583 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:57:51.407871 3583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f11607f00fcc42409b7523438c348102: Generated new TSK 0
I20260808 06:57:51.408051 3583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:57:51.415061 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44681
--builtin_ntp_servers=127.21.28.84:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 06:57:51.495769 3596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:51.495968 3596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:51.495997 3596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:51.497151 3596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:51.497191 3596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:51.497224 3596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:51.498546 3596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:51.498772 3596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:51.499046 3596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:51.503847 3616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.503875 3613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.503903 3608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:51.503993 3596 server_base.cc:1034] running on GCE node
I20260808 06:57:51.504259 3596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:51.504493 3596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:51.505630 3596 hybrid_clock.cc:648] HybridClock initialized: now 1786172271505612 us; error 31 us; skew 500 ppm
I20260808 06:57:51.510329 3596 webserver.cc:460] Webserver started at http://127.21.28.65:42323/ using document root <none> and password file <none>
I20260808 06:57:51.510573 3596 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:51.510624 3596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:51.510752 3596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:51.511861 3596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "cc04f904e5fd4bc0a3da9c194fb10c64"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "7450f125c79d0cc787eae41232253b76"
server_key_iv: "997a3e1966bf9e2eaf534c01673e3b4b"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.512189 3596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "cc04f904e5fd4bc0a3da9c194fb10c64"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "7450f125c79d0cc787eae41232253b76"
server_key_iv: "997a3e1966bf9e2eaf534c01673e3b4b"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.513392 3596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:57:51.514024 3625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.514174 3596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:51.514259 3596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "cc04f904e5fd4bc0a3da9c194fb10c64"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "7450f125c79d0cc787eae41232253b76"
server_key_iv: "997a3e1966bf9e2eaf534c01673e3b4b"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.514328 3596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:51.517665 3596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:51.517897 3596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:51.518010 3596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:51.518193 3596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:51.518481 3596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:51.518514 3596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.518541 3596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:51.518565 3596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.524121 3596 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:40793
I20260808 06:57:51.524161 3809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:40793 every 8 connection(s)
I20260808 06:57:51.524458 3596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:51.527105 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.525123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:59513 (local address 127.21.28.126:44681)
0808 06:57:51.525188 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:51.525190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:51.525377 (+ 187us) server_negotiation.cc:407] Connection header received
0808 06:57:51.525817 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:51.525818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:51.525824 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:51.525884 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:51.526919 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:51.526920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:51.526920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:51.526924 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:51.526937 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:51.526941 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:51.526987 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:51.527041 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 06:57:51.527053 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 06:57:51.527274 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.525088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:44681 (local address 127.21.28.65:59513)
0808 06:57:51.525261 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:51.525298 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:51.525746 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:51.525938 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:51.525941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:51.525949 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:51.526785 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:51.526785 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:51.526812 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:51.526840 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:51.526992 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:51.526995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:51.526997 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:51.527047 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 06:57:51.527093 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260808 06:57:51.527725 3811 heartbeater.cc:347] Connected to a master server at 127.21.28.126:44681
I20260808 06:57:51.527809 3811 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:51.527923 3811 heartbeater.cc:511] Master 127.21.28.126:44681 requested a full tablet report, sending...
I20260808 06:57:51.528403 3501 ts_manager.cc:194] Registered new tserver with Master: cc04f904e5fd4bc0a3da9c194fb10c64 (127.21.28.65:40793)
I20260808 06:57:51.530194 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 3596
I20260808 06:57:51.530267 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:51.530424 21617 external_mini_cluster.cc:1428] Setting key 5e7adb0fedb726edadc0ce38180f115c
I20260808 06:57:51.531193 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44681
--builtin_ntp_servers=127.21.28.84:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 06:57:51.610325 3815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:51.610519 3815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:51.610548 3815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:51.611708 3815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:51.611749 3815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:51.611783 3815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:51.613104 3815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:51.613336 3815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:51.613592 3815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:51.618232 3826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.618283 3823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.618337 3822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:51.618362 3815 server_base.cc:1034] running on GCE node
I20260808 06:57:51.618562 3815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:51.618779 3815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:51.619922 3815 hybrid_clock.cc:648] HybridClock initialized: now 1786172271619907 us; error 26 us; skew 500 ppm
I20260808 06:57:51.621049 3815 webserver.cc:460] Webserver started at http://127.21.28.66:42853/ using document root <none> and password file <none>
I20260808 06:57:51.621264 3815 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:51.621310 3815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:51.621438 3815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:51.622407 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "ab0188cb4d704b95b3af265277cba62e"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "81fe867891e5470127dc4c05a83e572e"
server_key_iv: "6b23eff5ee5ac39d28aafe2ddb6545d8"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.622725 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "ab0188cb4d704b95b3af265277cba62e"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "81fe867891e5470127dc4c05a83e572e"
server_key_iv: "6b23eff5ee5ac39d28aafe2ddb6545d8"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.623944 3815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:51.624569 3837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.624720 3815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:51.624802 3815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "ab0188cb4d704b95b3af265277cba62e"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "81fe867891e5470127dc4c05a83e572e"
server_key_iv: "6b23eff5ee5ac39d28aafe2ddb6545d8"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.624867 3815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:51.628062 3815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:51.628289 3815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:51.628407 3815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:51.628588 3815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:51.628865 3815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:51.628912 3815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.628937 3815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:51.628963 3815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.634554 3815 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:43415
I20260808 06:57:51.634595 4026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:43415 every 8 connection(s)
I20260808 06:57:51.634889 3815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:51.635710 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 3815
I20260808 06:57:51.635814 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:51.636039 21617 external_mini_cluster.cc:1428] Setting key abd4ac52bbcf6d2b0df6662f82147d04
I20260808 06:57:51.637064 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44681
--builtin_ntp_servers=127.21.28.84:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 06:57:51.638065 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.635542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:49317 (local address 127.21.28.126:44681)
0808 06:57:51.635598 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:51.635601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:51.635858 (+ 257us) server_negotiation.cc:407] Connection header received
0808 06:57:51.636467 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:51.636468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:51.636475 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:51.636530 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:51.637824 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:51.637826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:51.637826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:51.637830 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:51.637848 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:51.637852 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:51.637913 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:51.637995 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 06:57:51.638008 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:51.638267 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.635517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:44681 (local address 127.21.28.66:49317)
0808 06:57:51.635732 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:51.635777 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:51.636356 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:51.636582 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:51.636585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:51.636593 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:51.637633 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:51.637633 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:51.637674 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:51.637711 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:51.637921 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:51.637925 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:51.637928 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:51.638005 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 06:57:51.638066 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260808 06:57:51.638857 4028 heartbeater.cc:347] Connected to a master server at 127.21.28.126:44681
I20260808 06:57:51.638942 4028 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:51.639078 4028 heartbeater.cc:511] Master 127.21.28.126:44681 requested a full tablet report, sending...
I20260808 06:57:51.639424 3501 ts_manager.cc:194] Registered new tserver with Master: ab0188cb4d704b95b3af265277cba62e (127.21.28.66:43415)
W20260808 06:57:51.720577 4032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:51.720767 4032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:51.720788 4032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:51.721871 4032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:51.721916 4032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:51.721942 4032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:51.723203 4032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:51.723372 4032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:51.723596 4032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:51.728235 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.728255 4042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:51.728293 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:51.728371 4032 server_base.cc:1034] running on GCE node
I20260808 06:57:51.728875 4032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:51.729108 4032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:51.730239 4032 hybrid_clock.cc:648] HybridClock initialized: now 1786172271730227 us; error 24 us; skew 500 ppm
I20260808 06:57:51.731419 4032 webserver.cc:460] Webserver started at http://127.21.28.67:43789/ using document root <none> and password file <none>
I20260808 06:57:51.731623 4032 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:51.731693 4032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:51.731838 4032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:51.732867 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "3ea88a980776468e8cafc4a36ba800d5"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "47cfcbe6ba9960be05081f05435ab56d"
server_key_iv: "2e7c993ee4c75f380d22f3f9bb342b7d"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.733192 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "3ea88a980776468e8cafc4a36ba800d5"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "47cfcbe6ba9960be05081f05435ab56d"
server_key_iv: "2e7c993ee4c75f380d22f3f9bb342b7d"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.734359 4032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 06:57:51.734994 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.735143 4032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:51.735225 4032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "3ea88a980776468e8cafc4a36ba800d5"
format_stamp: "Formatted at 2026-08-08 06:57:51 on dist-test-slave-nj21"
server_key: "47cfcbe6ba9960be05081f05435ab56d"
server_key_iv: "2e7c993ee4c75f380d22f3f9bb342b7d"
server_key_version: "encryptionkey@0"
I20260808 06:57:51.735291 4032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:51.738480 4032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:51.738714 4032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:51.738840 4032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:51.739030 4032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:51.739308 4032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:51.739341 4032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.739367 4032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:51.739390 4032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:51.745159 4032 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:42283
I20260808 06:57:51.745201 4222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:42283 every 8 connection(s)
I20260808 06:57:51.745467 4032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:51.748059 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.746098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:54709 (local address 127.21.28.126:44681)
0808 06:57:51.746151 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:51.746153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:51.746360 (+ 207us) server_negotiation.cc:407] Connection header received
0808 06:57:51.746787 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:51.746789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:51.746794 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:51.746834 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:51.747890 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:51.747891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:51.747891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:57:51.747895 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:51.747909 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:57:51.747912 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:51.747947 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:51.747996 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 06:57:51.748007 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:57:51.748217 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:51.746080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:44681 (local address 127.21.28.67:54709)
0808 06:57:51.746252 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:51.746281 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:51.746700 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:51.746912 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:51.746915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:51.746923 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:51.747759 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:57:51.747760 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:51.747786 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:57:51.747811 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:51.747951 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:51.747954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:51.747956 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:57:51.748001 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 06:57:51.748044 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260808 06:57:51.748653 4223 heartbeater.cc:347] Connected to a master server at 127.21.28.126:44681
I20260808 06:57:51.748735 4223 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:51.748872 4223 heartbeater.cc:511] Master 127.21.28.126:44681 requested a full tablet report, sending...
I20260808 06:57:51.749189 3501 ts_manager.cc:194] Registered new tserver with Master: 3ea88a980776468e8cafc4a36ba800d5 (127.21.28.67:42283)
I20260808 06:57:51.752328 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 4032
I20260808 06:57:51.752400 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:51.752590 21617 external_mini_cluster.cc:1428] Setting key 6de5e1cc90b34a942f22352f69709f47
I20260808 06:57:51.753337 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 06:57:51.754969 3570 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:57:51.754607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.21.28.126:44681)
0808 06:57:51.754642 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:51.754644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:51.754737 (+ 93us) server_negotiation.cc:407] Connection header received
0808 06:57:51.754788 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:51.754789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:51.754792 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:51.754834 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:51.754924 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39340: BlockingRecv error: recv got EOF from 127.0.0.1:39340 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 06:57:51.755057 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 3596
I20260808 06:57:51.771661 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 3815
I20260808 06:57:51.800823 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 4032
I20260808 06:57:51.809742 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 3454
2026-08-08T06:57:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (577 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:51 dist-test-slave-nj21 krb5kdc[4240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:51 dist-test-slave-nj21 krb5kdc[4240](info): set up 2 sockets
Aug 08 06:57:51 dist-test-slave-nj21 krb5kdc[4240](info): commencing operation
krb5kdc: starting...
W20260808 06:57:52.938860 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:57:53.059422 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:45671
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:45671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:53.138978 4259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:53.139178 4259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:53.139209 4259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:53.140359 4259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:57:53.140399 4259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:53.140419 4259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:53.140434 4259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:57:53.140450 4259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:57:53.142303 4259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:45671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:45671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:53.142546 4259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:53.142750 4259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:53.147436 4267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:53.147444 4269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:53.147491 4266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:53.147543 4259 server_base.cc:1034] running on GCE node
I20260808 06:57:53.147974 4259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:53.148258 4259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:53.149413 4259 hybrid_clock.cc:648] HybridClock initialized: now 1786172273149399 us; error 30 us; skew 500 ppm
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:53.151969 4259 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:57:53.152395 4259 webserver.cc:460] Webserver started at http://127.21.28.126:37957/ using document root <none> and password file <none>
I20260808 06:57:53.152594 4259 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:53.152647 4259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:53.152796 4259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:53.153950 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "4b0931015a174997b4d536435afab879"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "9449e1ec9690024182ad2b087816bfbd"
server_key_iv: "1a1486c9f17231dd8dca742c35a157fd"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.154331 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "4b0931015a174997b4d536435afab879"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "9449e1ec9690024182ad2b087816bfbd"
server_key_iv: "1a1486c9f17231dd8dca742c35a157fd"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.155712 4259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 06:57:53.156435 4276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.156589 4259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:53.156673 4259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "4b0931015a174997b4d536435afab879"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "9449e1ec9690024182ad2b087816bfbd"
server_key_iv: "1a1486c9f17231dd8dca742c35a157fd"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.156764 4259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:53.169776 4259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:53.171342 4259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:53.171464 4259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:53.175114 4259 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:45671
I20260808 06:57:53.175165 4371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:45671 every 8 connection(s)
I20260808 06:57:53.175444 4259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:57:53.176352 4372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:53.178534 4372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879: Bootstrap starting.
I20260808 06:57:53.179060 4372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:53.179312 4372 log.cc:824] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:53.179987 4372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879: No bootstrap required, opened a new log
I20260808 06:57:53.181542 4372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0931015a174997b4d536435afab879" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45671 } }
I20260808 06:57:53.181669 4372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:53.181702 4372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0931015a174997b4d536435afab879, State: Initialized, Role: FOLLOWER
I20260808 06:57:53.181834 4372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0931015a174997b4d536435afab879" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45671 } }
I20260808 06:57:53.181896 4372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:57:53.181926 4372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:57:53.181972 4372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:53.182721 4372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0931015a174997b4d536435afab879" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45671 } }
I20260808 06:57:53.182823 4372 leader_election.cc:302] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b0931015a174997b4d536435afab879; no voters:
I20260808 06:57:53.182974 4372 leader_election.cc:288] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:57:53.183012 4378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:53.183151 4378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0931015a174997b4d536435afab879, State: Running, Role: LEADER
I20260808 06:57:53.183203 4372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:57:53.183249 4378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0931015a174997b4d536435afab879" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45671 } }
I20260808 06:57:53.183590 4380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b0931015a174997b4d536435afab879. Latest consensus state: current_term: 1 leader_uuid: "4b0931015a174997b4d536435afab879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0931015a174997b4d536435afab879" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45671 } } }
I20260808 06:57:53.183787 4380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:53.183882 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 4259
I20260808 06:57:53.183995 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:57:53.184139 4391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:57:53.184185 21617 external_mini_cluster.cc:1428] Setting key be63cbc6bcba286ba8870122523c9597
I20260808 06:57:53.183570 4379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b0931015a174997b4d536435afab879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0931015a174997b4d536435afab879" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45671 } } }
I20260808 06:57:53.184521 4379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879 [sys.catalog]: This master's current role is: LEADER
I20260808 06:57:53.184626 4391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:57:53.186132 4391 catalog_manager.cc:1347] Generated new cluster ID: f7bb7dbb50a8437788967a6549ffafcd
I20260808 06:57:53.186182 4391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:57:53.193876 4391 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:57:53.194448 4391 catalog_manager.cc:1504] Loading token signing keys...
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:53.199937 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.184630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.21.28.126:45671)
0808 06:57:53.184953 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.184985 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.185017 (+ 32us) server_negotiation.cc:407] Connection header received
0808 06:57:53.185192 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.185202 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.185263 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.185390 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:53.190328 (+ 4938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.192220 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.193274 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.193879 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.193990 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.196738 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:53.196742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:53.196743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:53.196754 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:53.198455 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.198832 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.198835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.198835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.198934 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.199113 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.199114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.199114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.199342 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:53.199466 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.199564 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.199726 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":72,"threads_started":1}
I20260808 06:57:53.200918 4391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879: Generated new TSK 0
I20260808 06:57:53.201087 4391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:57:53.228425 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45671
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:53.308718 4411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:53.308928 4411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:53.308961 4411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:53.310102 4411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:53.310142 4411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:53.310176 4411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:57:53.311579 4411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:53.311830 4411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:53.312041 4411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:53.316871 4420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:53.316916 4423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:53.316843 4411 server_base.cc:1034] running on GCE node
W20260808 06:57:53.316865 4418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:53.317204 4411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:53.317412 4411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:53.318553 4411 hybrid_clock.cc:648] HybridClock initialized: now 1786172273318538 us; error 36 us; skew 500 ppm
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:53.321003 4411 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:57:53.321408 4411 webserver.cc:460] Webserver started at http://127.21.28.65:39381/ using document root <none> and password file <none>
I20260808 06:57:53.321593 4411 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:53.321632 4411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:53.321722 4411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:53.322652 4411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "c23e71e86bab4c18a93a081d66599b1e"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "10953ea4499389c399d8600d0c173db2"
server_key_iv: "88554594cd03a3e390cddea8acc63b6a"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.322939 4411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "c23e71e86bab4c18a93a081d66599b1e"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "10953ea4499389c399d8600d0c173db2"
server_key_iv: "88554594cd03a3e390cddea8acc63b6a"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.324127 4411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:57:53.324765 4436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.324918 4411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:57:53.324995 4411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "c23e71e86bab4c18a93a081d66599b1e"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "10953ea4499389c399d8600d0c173db2"
server_key_iv: "88554594cd03a3e390cddea8acc63b6a"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.325047 4411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:53.344722 4411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:53.346498 4411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:53.346611 4411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:53.346779 4411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:53.347054 4411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:53.347084 4411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.347105 4411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:53.347121 4411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.352814 4411 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:46437
I20260808 06:57:53.352859 4627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:46437 every 8 connection(s)
I20260808 06:57:53.353155 4411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:57:53.353535 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 4411
I20260808 06:57:53.353618 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:57:53.353828 21617 external_mini_cluster.cc:1428] Setting key 3abf148e63b9a3e9b3f24a27263d1798
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
I20260808 06:57:53.362641 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.353771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:60409 (local address 127.21.28.126:45671)
0808 06:57:53.353820 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.353822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.354104 (+ 282us) server_negotiation.cc:407] Connection header received
0808 06:57:53.354762 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.354762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.354772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.354823 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:53.355991 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.356673 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.357601 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.357822 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.357876 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.360133 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:53.360134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:53.360134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:53.360142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:53.361415 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.361768 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.361769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.361769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.361828 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.362046 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.362047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.362047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.362154 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:53.362209 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.362461 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.362552 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:57:53.362784 4632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.353755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45671 (local address 127.21.28.65:60409)
0808 06:57:53.353967 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.354013 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:53.354666 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:53.354893 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:53.354897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:53.355624 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:53.355914 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.355921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.356758 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.356759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.357528 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.357530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.357929 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.357930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.358071 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.358502 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:53.358511 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:53.360047 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:53.361488 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:53.361494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:53.361496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:53.361704 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:53.361881 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:53.361882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:53.361882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:53.361992 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:53.362208 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:53.362214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:53.362342 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 06:57:53.362426 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 06:57:53.362576 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260808 06:57:53.363227 4628 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45671
I20260808 06:57:53.363314 4628 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:53.363516 4628 heartbeater.cc:511] Master 127.21.28.126:45671 requested a full tablet report, sending...
I20260808 06:57:53.363999 4306 ts_manager.cc:194] Registered new tserver with Master: c23e71e86bab4c18a93a081d66599b1e (127.21.28.65:46437)
I20260808 06:57:53.364601 4306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:60409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:57:53.378751 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45671
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:53.458429 4638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:53.458618 4638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:53.458647 4638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:53.459810 4638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:53.459849 4638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:53.459883 4638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:57:53.461290 4638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:53.461527 4638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:53.461716 4638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:53.466702 4654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:53.466706 4648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:53.466882 4649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:53.467064 4638 server_base.cc:1034] running on GCE node
I20260808 06:57:53.467430 4638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:53.468454 4638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:53.469659 4638 hybrid_clock.cc:648] HybridClock initialized: now 1786172273469589 us; error 32 us; skew 500 ppm
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:53.475967 4638 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:57:53.477468 4638 webserver.cc:460] Webserver started at http://127.21.28.66:39387/ using document root <none> and password file <none>
I20260808 06:57:53.477856 4638 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:53.477906 4638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:53.478029 4638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:53.480185 4638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "3ca858bf5f784d3f94bd096169e609ac"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "589da853efa1242aaef330bc6ea25d44"
server_key_iv: "19c966cecfa2e0b92a7bbd4d66c656ba"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.480755 4638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "3ca858bf5f784d3f94bd096169e609ac"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "589da853efa1242aaef330bc6ea25d44"
server_key_iv: "19c966cecfa2e0b92a7bbd4d66c656ba"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.482151 4638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:57:53.482805 4662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.482955 4638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:53.483055 4638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "3ca858bf5f784d3f94bd096169e609ac"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "589da853efa1242aaef330bc6ea25d44"
server_key_iv: "19c966cecfa2e0b92a7bbd4d66c656ba"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.483124 4638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:53.493824 4638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:53.495834 4638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:53.495972 4638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:53.496166 4638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:53.496439 4638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:53.496469 4638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.496487 4638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:53.496503 4638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.502243 4638 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40481
I20260808 06:57:53.502300 4843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40481 every 8 connection(s)
I20260808 06:57:53.502564 4638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:57:53.503273 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 4638
I20260808 06:57:53.503360 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:57:53.503583 21617 external_mini_cluster.cc:1428] Setting key 72b78279c58b0e0084d91a964488776e
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:53.511785 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.503253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:38067 (local address 127.21.28.126:45671)
0808 06:57:53.503306 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.503309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.503560 (+ 251us) server_negotiation.cc:407] Connection header received
0808 06:57:53.504177 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.504179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.504190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.504248 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:53.505405 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.505945 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.506886 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.507091 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.507143 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.509258 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:53.509259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:53.509259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:53.509266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:53.510464 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.510897 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.510898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.510898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.510942 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.511183 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.511183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.511184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.511279 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:53.511324 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.511618 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.511696 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 06:57:53.512063 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.503196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45671 (local address 127.21.28.66:38067)
0808 06:57:53.503422 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.503469 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:53.504081 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:53.504324 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:53.504327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:53.505057 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:53.505314 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.505319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.506033 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.506034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.506804 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.506806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.507152 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.507153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.507245 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.507700 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:53.507709 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:53.509169 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:53.510542 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:53.510550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:53.510553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:53.510829 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:53.510992 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:53.510993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:53.510993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:53.511121 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:53.511333 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:53.511338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:53.511496 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 06:57:53.511610 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 06:57:53.511846 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260808 06:57:53.512540 4844 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45671
I20260808 06:57:53.512626 4844 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
I20260808 06:57:53.512784 4844 heartbeater.cc:511] Master 127.21.28.126:45671 requested a full tablet report, sending...
I20260808 06:57:53.513111 4306 ts_manager.cc:194] Registered new tserver with Master: 3ca858bf5f784d3f94bd096169e609ac (127.21.28.66:40481)
I20260808 06:57:53.513530 4306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:38067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:57:53.528565 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45671
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:57:53.608780 4851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:57:53.608974 4851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:57:53.609006 4851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:57:53.610141 4851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:57:53.610180 4851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:57:53.610214 4851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:57:53.611621 4851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:57:53.611883 4851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:57:53.612087 4851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:57:53.616765 4863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:53.616845 4861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:57:53.616765 4859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:57:53.616979 4851 server_base.cc:1034] running on GCE node
I20260808 06:57:53.617166 4851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:57:53.617373 4851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:57:53.618508 4851 hybrid_clock.cc:648] HybridClock initialized: now 1786172273618492 us; error 29 us; skew 500 ppm
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:57:53.620905 4851 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:57:53.621338 4851 webserver.cc:460] Webserver started at http://127.21.28.67:44493/ using document root <none> and password file <none>
I20260808 06:57:53.621556 4851 fs_manager.cc:359] Metadata directory not provided
I20260808 06:57:53.621608 4851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:57:53.621747 4851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:57:53.622814 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "95f26990e37f4b7d8c7ea19cf28edc6d"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "8ac3462b45e5b6a4d2e7cf1f23033846"
server_key_iv: "738c69a0440b010b85871d95aa06f763"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.623147 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "95f26990e37f4b7d8c7ea19cf28edc6d"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "8ac3462b45e5b6a4d2e7cf1f23033846"
server_key_iv: "738c69a0440b010b85871d95aa06f763"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.624528 4851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:57:53.625252 4873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.625406 4851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:57:53.625496 4851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "95f26990e37f4b7d8c7ea19cf28edc6d"
format_stamp: "Formatted at 2026-08-08 06:57:53 on dist-test-slave-nj21"
server_key: "8ac3462b45e5b6a4d2e7cf1f23033846"
server_key_iv: "738c69a0440b010b85871d95aa06f763"
server_key_version: "encryptionkey@0"
I20260808 06:57:53.625558 4851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:57:53.648981 4851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:57:53.651000 4851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:57:53.651121 4851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:57:53.651322 4851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:57:53.651610 4851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:57:53.651643 4851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.651695 4851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:57:53.651723 4851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:57:53.657447 4851 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:36087
I20260808 06:57:53.657518 5050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:36087 every 8 connection(s)
I20260808 06:57:53.657806 4851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:57:53.663113 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 4851
I20260808 06:57:53.663213 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:57:53.663385 21617 external_mini_cluster.cc:1428] Setting key a0e96c016fcf9c8ef8cde5350929126c
Aug 08 06:57:53 dist-test-slave-nj21 krb5kdc[4240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:57:53.666357 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.658553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:43831 (local address 127.21.28.126:45671)
0808 06:57:53.658640 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.658642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.658795 (+ 153us) server_negotiation.cc:407] Connection header received
0808 06:57:53.659312 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.659313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.659322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.659371 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:53.660316 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.660798 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.661741 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.661934 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.661984 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.664224 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:53.664225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:53.664226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:53.664233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:53.665332 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.665668 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.665669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.665669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.665721 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.665895 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.665896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.665896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.665991 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:53.666041 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.666232 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.666293 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:57:53.666523 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.658482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45671 (local address 127.21.28.67:43831)
0808 06:57:53.658666 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.658725 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:53.659229 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:53.659423 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:53.659426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:53.660026 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:57:53.660224 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.660230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.660863 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.660863 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.661662 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.661665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.662000 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.662001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.662120 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.662568 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:57:53.662575 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:57:53.664137 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:57:53.665399 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:53.665404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:53.665406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:53.665611 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:53.665767 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:57:53.665767 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:57:53.665767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:57:53.665852 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:57:53.666035 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:57:53.666038 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:57:53.666139 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 06:57:53.666210 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 06:57:53.666329 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260808 06:57:53.666952 5052 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45671
I20260808 06:57:53.667039 5052 heartbeater.cc:464] Registering TS with master...
I20260808 06:57:53.667181 5052 heartbeater.cc:511] Master 127.21.28.126:45671 requested a full tablet report, sending...
I20260808 06:57:53.667507 4306 ts_manager.cc:194] Registered new tserver with Master: 95f26990e37f4b7d8c7ea19cf28edc6d (127.21.28.67:36087)
I20260808 06:57:53.667920 4306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:43831
I20260808 06:57:53.675184 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:57:53.684558 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.676737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.21.28.126:45671)
0808 06:57:53.676783 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.676785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.676999 (+ 214us) server_negotiation.cc:407] Connection header received
0808 06:57:53.677033 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.677034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.677046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.677121 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:57:53.677787 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.678474 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.679398 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.679720 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.679794 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.680986 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:57:53.680988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:57:53.680988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:57:53.680996 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:57:53.682991 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.683381 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.683382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.683382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.683447 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:57:53.683783 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:57:53.683784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:57:53.683784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:57:53.683939 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:57:53.684039 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.684386 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.684463 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:53.686407 4306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:57:53.688378 4306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:57:53.694482 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.690496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:36087 (local address 127.0.0.1:46560)
0808 06:57:53.690928 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.690940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:53.691007 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:53.691244 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:53.691245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:53.691246 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:53.691287 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.691288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.692340 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.692341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.693229 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.693232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.694160 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.694161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.694254 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.694273 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 06:57:53.694345 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 06:57:53.694402 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":40,"thread_start_us":57,"threads_started":1}
I20260808 06:57:53.694576 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.690780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.21.28.66:40481)
0808 06:57:53.690945 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.690949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.691241 (+ 292us) server_negotiation.cc:407] Connection header received
0808 06:57:53.691290 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.691292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.691308 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.691394 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:53.691540 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.692301 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.693668 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.694218 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.694291 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.694352 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.694452 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.694495 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260808 06:57:53.694572 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.690496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40481 (local address 127.0.0.1:35026)
0808 06:57:53.691099 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.691129 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:53.691181 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:53.691440 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:53.691440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:53.691442 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:53.691482 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.691484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.692407 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.692409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.693572 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.693575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.694306 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.694307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.694376 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.694380 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:53.694420 (+ 40us) client_negotiation.cc:240] Negotiation successful
0808 06:57:53.694456 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":27,"thread_start_us":58,"threads_started":1}
I20260808 06:57:53.694976 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.690759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.21.28.67:36087)
0808 06:57:53.690944 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.690949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.690993 (+ 44us) server_negotiation.cc:407] Connection header received
0808 06:57:53.691084 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.691085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.691097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.691164 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:53.691363 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.692240 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.693348 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.694067 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.694140 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.694192 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.694857 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.694901 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260808 06:57:53.695386 4744 tablet_service.cc:1467] Processing CreateTablet for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0 (DEFAULT_TABLE table=test-table [id=dedf4d62f642409298adee757e435fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:53.695582 4949 tablet_service.cc:1467] Processing CreateTablet for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0 (DEFAULT_TABLE table=test-table [id=dedf4d62f642409298adee757e435fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:53.695802 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:53.695955 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.690493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:46437 (local address 127.0.0.1:39072)
0808 06:57:53.690672 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.690717 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:53.690786 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:53.691207 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:53.691211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:53.691213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:53.691277 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.691279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.693619 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.693620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.694833 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.694836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.695753 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.695754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.695823 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.695827 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:57:53.695857 (+ 30us) client_negotiation.cc:240] Negotiation successful
0808 06:57:53.695892 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260808 06:57:53.695966 4949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:53.696321 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.690592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.21.28.65:46437)
0808 06:57:53.690990 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.690995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.691007 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:57:53.691051 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.691053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.691069 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.691160 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:53.692355 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.693477 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.694918 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.695630 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.696120 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.696180 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.696204 (+ 24us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.696244 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":98,"threads_started":1}
I20260808 06:57:53.698520 5071 tablet_bootstrap.cc:492] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Bootstrap starting.
I20260808 06:57:53.698989 4523 tablet_service.cc:1467] Processing CreateTablet for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0 (DEFAULT_TABLE table=test-table [id=dedf4d62f642409298adee757e435fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:57:53.699120 5072 tablet_bootstrap.cc:492] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Bootstrap starting.
I20260808 06:57:53.699291 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:57:53.699411 5071 tablet_bootstrap.cc:654] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:53.699895 5072 tablet_bootstrap.cc:654] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:53.700040 5071 log.cc:824] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:53.700582 5072 log.cc:824] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:53.700953 5071 tablet_bootstrap.cc:492] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: No bootstrap required, opened a new log
I20260808 06:57:53.701020 5071 ts_tablet_manager.cc:1397] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 06:57:53.701406 5072 tablet_bootstrap.cc:492] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: No bootstrap required, opened a new log
I20260808 06:57:53.701480 5072 ts_tablet_manager.cc:1397] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:57:53.702196 5079 tablet_bootstrap.cc:492] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e: Bootstrap starting.
I20260808 06:57:53.702706 5079 tablet_bootstrap.cc:654] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e: Neither blocks nor log segments found. Creating new log.
I20260808 06:57:53.702989 5071 raft_consensus.cc:348] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.703161 5071 raft_consensus.cc:374] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:53.703168 5079 log.cc:824] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:57:53.703204 5071 raft_consensus.cc:729] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca858bf5f784d3f94bd096169e609ac, State: Initialized, Role: FOLLOWER
I20260808 06:57:53.703346 5071 consensus_queue.cc:260] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.703626 5071 ts_tablet_manager.cc:1428] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 06:57:53.703768 4844 heartbeater.cc:503] Master 127.21.28.126:45671 was elected leader, sending a full tablet report...
I20260808 06:57:53.703657 5072 raft_consensus.cc:348] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.703836 5072 raft_consensus.cc:374] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:53.703876 5072 raft_consensus.cc:729] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f26990e37f4b7d8c7ea19cf28edc6d, State: Initialized, Role: FOLLOWER
I20260808 06:57:53.703886 5079 tablet_bootstrap.cc:492] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e: No bootstrap required, opened a new log
I20260808 06:57:53.703943 5079 ts_tablet_manager.cc:1397] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:57:53.703982 5072 consensus_queue.cc:260] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.704241 5072 ts_tablet_manager.cc:1428] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 06:57:53.704254 5052 heartbeater.cc:503] Master 127.21.28.126:45671 was elected leader, sending a full tablet report...
I20260808 06:57:53.705585 5079 raft_consensus.cc:348] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.705709 5079 raft_consensus.cc:374] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:57:53.705740 5079 raft_consensus.cc:729] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23e71e86bab4c18a93a081d66599b1e, State: Initialized, Role: FOLLOWER
I20260808 06:57:53.705847 5079 consensus_queue.cc:260] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.706050 5079 ts_tablet_manager.cc:1428] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 06:57:53.706106 4628 heartbeater.cc:503] Master 127.21.28.126:45671 was elected leader, sending a full tablet report...
W20260808 06:57:53.753089 4846 tablet.cc:2369] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:57:53.853863 4631 tablet.cc:2369] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:53.879762 5085 raft_consensus.cc:482] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:57:53.879917 5085 raft_consensus.cc:504] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.880291 5085 leader_election.cc:288] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c23e71e86bab4c18a93a081d66599b1e (127.21.28.65:46437), 3ca858bf5f784d3f94bd096169e609ac (127.21.28.66:40481)
I20260808 06:57:53.882926 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.880484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:46437 (local address 127.21.28.67:40777)
0808 06:57:53.880540 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.880551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:53.880592 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:53.880755 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:53.880756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:53.880758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:53.880821 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.880822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.881435 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.881435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.882178 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.882180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.882685 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.882686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.882763 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.882766 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:53.882809 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 06:57:53.882849 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 06:57:53.882927 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.880554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:40777 (local address 127.21.28.65:46437)
0808 06:57:53.880601 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.880603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.880617 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:57:53.880649 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.880650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.880663 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.880723 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:53.880880 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.881354 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.882266 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.882621 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.882682 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.882691 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.882821 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.882856 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 06:57:53.883046 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.880602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:39357 (local address 127.21.28.66:40481)
0808 06:57:53.880646 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.880648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.880755 (+ 107us) server_negotiation.cc:407] Connection header received
0808 06:57:53.880835 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.880837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.880847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.880892 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:57:53.881039 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.881492 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.882377 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.882748 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.882813 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.882822 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.882945 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.882981 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:57:53.883090 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.880492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40481 (local address 127.21.28.67:39357)
0808 06:57:53.880690 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.880698 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:57:53.880742 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:57:53.880902 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:57:53.880903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:57:53.880904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:57:53.880978 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.880980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.881562 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.881562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.882295 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:57:53.882297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.882819 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.882820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:57:53.882886 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.882889 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:57:53.882933 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 06:57:53.882968 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":58,"thread_start_us":99,"threads_started":1}
I20260808 06:57:53.883303 4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a5b2c7c8534cb381af6a3e56cfb8a0" candidate_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e71e86bab4c18a93a081d66599b1e" is_pre_election: true
I20260808 06:57:53.883419 4779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a5b2c7c8534cb381af6a3e56cfb8a0" candidate_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca858bf5f784d3f94bd096169e609ac" is_pre_election: true
I20260808 06:57:53.883490 4556 raft_consensus.cc:2454] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95f26990e37f4b7d8c7ea19cf28edc6d in term 0.
I20260808 06:57:53.883584 4779 raft_consensus.cc:2454] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95f26990e37f4b7d8c7ea19cf28edc6d in term 0.
I20260808 06:57:53.883750 4878 leader_election.cc:302] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f26990e37f4b7d8c7ea19cf28edc6d, c23e71e86bab4c18a93a081d66599b1e; no voters:
I20260808 06:57:53.883893 5085 raft_consensus.cc:2781] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:57:53.883960 5085 raft_consensus.cc:482] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:57:53.883977 5085 raft_consensus.cc:3037] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:53.884790 5085 raft_consensus.cc:504] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.884909 5085 leader_election.cc:288] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [CANDIDATE]: Term 1 election: Requested vote from peers c23e71e86bab4c18a93a081d66599b1e (127.21.28.65:46437), 3ca858bf5f784d3f94bd096169e609ac (127.21.28.66:40481)
I20260808 06:57:53.885085 4779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a5b2c7c8534cb381af6a3e56cfb8a0" candidate_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca858bf5f784d3f94bd096169e609ac"
I20260808 06:57:53.885159 4779 raft_consensus.cc:3037] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:53.885943 4779 raft_consensus.cc:2454] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f26990e37f4b7d8c7ea19cf28edc6d in term 1.
I20260808 06:57:53.886075 4556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a5b2c7c8534cb381af6a3e56cfb8a0" candidate_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e71e86bab4c18a93a081d66599b1e"
I20260808 06:57:53.886111 4874 leader_election.cc:302] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca858bf5f784d3f94bd096169e609ac, 95f26990e37f4b7d8c7ea19cf28edc6d; no voters:
I20260808 06:57:53.886150 4556 raft_consensus.cc:3037] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:57:53.886188 5085 raft_consensus.cc:2781] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:57:53.886327 5085 raft_consensus.cc:686] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 1 LEADER]: Becoming Leader. State: Replica: 95f26990e37f4b7d8c7ea19cf28edc6d, State: Running, Role: LEADER
I20260808 06:57:53.886504 5085 consensus_queue.cc:237] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } }
I20260808 06:57:53.886950 4556 raft_consensus.cc:2454] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f26990e37f4b7d8c7ea19cf28edc6d in term 1.
I20260808 06:57:53.887372 4305 catalog_manager.cc:5495] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 95f26990e37f4b7d8c7ea19cf28edc6d (127.21.28.67). New cstate: current_term: 1 leader_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95f26990e37f4b7d8c7ea19cf28edc6d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36087 } health_report { overall_health: HEALTHY } } }
W20260808 06:57:53.908329 5053 tablet.cc:2369] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:57:53.918983 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.916749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.21.28.67:36087)
0808 06:57:53.916796 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 06:57:53.916798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:57:53.916806 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:57:53.916875 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:57:53.916876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:57:53.916893 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:57:53.916959 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:57:53.917103 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.917553 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.918284 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:57:53.918471 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:57:53.918526 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:57:53.918677 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:57:53.918768 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:57:53.918838 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 06:57:53.918876 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 06:57:53.918894 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:57:53.920528 4556 raft_consensus.cc:1261] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [term 1 FOLLOWER]: Refusing update from remote peer 95f26990e37f4b7d8c7ea19cf28edc6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:53.920527 4779 raft_consensus.cc:1261] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 1 FOLLOWER]: Refusing update from remote peer 95f26990e37f4b7d8c7ea19cf28edc6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:57:53.920845 5090 consensus_queue.cc:1035] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ca858bf5f784d3f94bd096169e609ac" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:53.920966 5085 consensus_queue.cc:1035] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c23e71e86bab4c18a93a081d66599b1e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 46437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:57:53.922930 5093 mvcc.cc:205] Tried to move back new op lower bound from 7316161633976201216 to 7316161633839734784. Current Snapshot: MvccSnapshot[applied={T|T < 7316161633976201216}]
I20260808 06:57:53.923847 5097 mvcc.cc:205] Tried to move back new op lower bound from 7316161633976201216 to 7316161633839734784. Current Snapshot: MvccSnapshot[applied={T|T < 7316161633976201216}]
I20260808 06:57:53.925019 4305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:53.925112 4305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:57:53.925793 4305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b0931015a174997b4d536435afab879: Sending DeleteTablet for 3 replicas of tablet b7a5b2c7c8534cb381af6a3e56cfb8a0
I20260808 06:57:53.926096 4744 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:53 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:35026
I20260808 06:57:53.926119 4523 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:53 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:39072
I20260808 06:57:53.926191 4949 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:57:53 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:46560
I20260808 06:57:53.926353 5113 tablet_replica.cc:330] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e: stopping tablet replica
I20260808 06:57:53.926452 5114 tablet_replica.cc:330] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: stopping tablet replica
I20260808 06:57:53.926542 5113 raft_consensus.cc:2229] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P c23e71e86bab4c18a93a081d66599b1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:53.926601 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 4411
I20260808 06:57:53.926541 5114 raft_consensus.cc:2229] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:57:53.933279 5114 raft_consensus.cc:2258] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:53.933756 5114 ts_tablet_manager.cc:1905] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:53.935432 5114 ts_tablet_manager.cc:1918] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:53.935514 5114 log.cc:1196] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/b7a5b2c7c8534cb381af6a3e56cfb8a0
I20260808 06:57:53.935860 5114 ts_tablet_manager.cc:1939] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 95f26990e37f4b7d8c7ea19cf28edc6d: Deleting consensus metadata
I20260808 06:57:53.936278 5110 tablet_replica.cc:330] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: stopping tablet replica
I20260808 06:57:53.936390 5110 raft_consensus.cc:2229] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:57:53.936496 5110 raft_consensus.cc:2258] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:57:53.936466 4284 catalog_manager.cc:4841] TS 95f26990e37f4b7d8c7ea19cf28edc6d (127.21.28.67:36087): tablet b7a5b2c7c8534cb381af6a3e56cfb8a0 (table test-table [id=dedf4d62f642409298adee757e435fb5]) successfully deleted
I20260808 06:57:53.937620 5110 ts_tablet_manager.cc:1905] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:57:53.939442 5110 ts_tablet_manager.cc:1918] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:57:53.939513 5110 log.cc:1196] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/b7a5b2c7c8534cb381af6a3e56cfb8a0
I20260808 06:57:53.939790 5110 ts_tablet_manager.cc:1939] T b7a5b2c7c8534cb381af6a3e56cfb8a0 P 3ca858bf5f784d3f94bd096169e609ac: Deleting consensus metadata
I20260808 06:57:53.940215 4280 catalog_manager.cc:4841] TS 3ca858bf5f784d3f94bd096169e609ac (127.21.28.66:40481): tablet b7a5b2c7c8534cb381af6a3e56cfb8a0 (table test-table [id=dedf4d62f642409298adee757e435fb5]) successfully deleted
W20260808 06:57:53.942837 4282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:46437 (error 108)
I20260808 06:57:53.943316 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:57:53.943186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:46437 (local address 127.0.0.1:39084)
0808 06:57:53.943242 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 06:57:53.943281 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:46437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 06:57:53.943392 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 4638
W20260808 06:57:53.947114 4282 catalog_manager.cc:4584] TS c23e71e86bab4c18a93a081d66599b1e (127.21.28.65:46437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7a5b2c7c8534cb381af6a3e56cfb8a0: Network error: Client connection negotiation failed: client connection to 127.21.28.65:46437: connect: Connection refused (error 111)
I20260808 06:57:53.951251 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 4851
I20260808 06:57:53.961731 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 4259
2026-08-08T06:57:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2152 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:57:54 dist-test-slave-nj21 krb5kdc[5122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:57:54 dist-test-slave-nj21 krb5kdc[5122](info): set up 2 sockets
Aug 08 06:57:54 dist-test-slave-nj21 krb5kdc[5122](info): commencing operation
krb5kdc: starting...
W20260808 06:57:55.079953 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:57:55 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:57:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.28.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.28.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/rangeradmin_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/rangeradmin_127.21.28.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.28.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.28.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/rangerlookup_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/rangerlookup_127.21.28.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.28.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.28.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/HTTP_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/HTTP_127.21.28.84@KRBTEST.COM.keytab.
I20260808 06:57:55.225808 21617 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDMxMqG/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/postgres -l logfile start
2026-08-08 06:57:57.433 UTC [5176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 06:57:57.433 UTC [5176] LOG: listening on IPv4 address "127.21.28.84", port 48041
2026-08-08 06:57:57.436 UTC [5176] LOG: listening on Unix socket "/tmp/.s.PGSQL.48041"
2026-08-08 06:57:57.456 UTC [5183] LOG: database system was shut down at 2026-08-08 06:57:56 UTC
2026-08-08 06:57:57.459 UTC [5176] LOG: database system is ready to accept connections
I20260808 06:57:58.513839 21617 mini_postgres.cc:96] Postgres bound to 48041
2026-08-08 06:57:58.521 UTC [5196] FATAL: database "slave" does not exist
127.21.28.84:48041 - accepting connections
I20260808 06:57:58.521803 21617 mini_ranger.cc:161] Starting Ranger...
I20260808 06:57:58.536123 21617 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 06:57:58.937564 21617 mini_ranger.cc:87] Created ranger Postgres database
I20260808 06:57:58.937644 21617 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-admin
2026-08-08 06:57:59,255 [I] DB FLAVOR :POSTGRES
2026-08-08 06:57:59,255 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 06:57:59,255 [I] Checking connection..
2026-08-08 06:57:59,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 06:57:59,495 [I] Checking connection passed.
2026-08-08 06:57:59,495 [I] --------- Verifying version history table ---------
2026-08-08 06:57:59,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:57:59,752 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 06:57:59,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:58:00,006 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 06:58:00,007 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 06:58:00,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 06:58:00.238 UTC [5293] WARNING: there is no transaction in progress
2026-08-08 06:58:00,246 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 06:58:00,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:58:00,501 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 06:58:00,501 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 06:58:00,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 06:58:00,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 06:58:01,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nj21.c.gcp-upstream.internal','N') ;"
2026-08-08 06:58:01,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 06:58:01,506 [I] Table x_portal_user does not exist in database ranger
2026-08-08 06:58:01,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 06:58:01,738 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 06:58:01,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 06:58:01,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 06:58:02,222 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 06:58:02,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 06:58:03.089 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.103 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.118 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.132 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.145 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.160 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.208 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.213 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.225 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.234 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.242 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.254 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.262 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.267 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.273 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03.278 UTC [5465] WARNING: there is no transaction in progress
2026-08-08 06:58:03,804 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 06:58:03,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 06:58:04,037 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 06:58:04,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 06:58:04,294 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 06:58:04,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 06:58:04,549 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 06:58:04,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 06:58:04,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 06:58:05,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nj21.c.gcp-upstream.internal';"
2026-08-08 06:58:05,258 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 06:58:05,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 06:58:05,509 [I] DB_PATCHES have already been applied
I20260808 06:58:05.514853 21617 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDMxMqG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 06:58:05.514914 21617 mini_ranger.cc:193] Using host: 127.21.28.84
I20260808 06:58:05.518133 21617 mini_ranger.cc:227] Ranger admin URL: http://127.21.28.84:39397
Aug 08, 2026 6:58:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:58:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:58:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 6:58:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 6:58:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 6:58:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 6:58:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 06:58:08.035338 21617 mini_ranger.cc:160] Time spent starting Ranger: real 9.514s user 0.001s sys 0.010s
Aug 08, 2026 6:58:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 6:58:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 6:58:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 6:58:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 6:58:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 06:58:28.347878 21617 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.28.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.28.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/rangerkms_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/rangerkms_127.21.28.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.28.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.28.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.28.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/HTTP_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.28.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/HTTP_127.21.28.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 06:58:28 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172308, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 06:58:28.424333 21617 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 06:58:28.436798 21617 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 06:58:29.183442 21617 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 06:58:29.183527 21617 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms
I20260808 06:58:29.183547 21617 mini_ranger_kms.cc:227] Using postgres at 127.21.28.84:48041
2026-08-08 06:58:29,826 [I] DB FLAVOR :POSTGRES
2026-08-08 06:58:29,826 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 06:58:29,826 [I] Checking connection
2026-08-08 06:58:29,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 06:58:30,177 [I] connection success
2026-08-08 06:58:30,177 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 06:58:30,178 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 06:58:30,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 06:58:30,417 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 06:58:30,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 06:58:30,417 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 06:58:30,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:48041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 06:58:30,676 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 06:58:31.088117 21617 mini_ranger_kms.cc:310] Created kms service
I20260808 06:58:31.333554 21617 mini_ranger_kms.cc:326] Created kudu user
I20260808 06:58:31.373092 21617 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 06:58:31 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172308, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:58:31.754634 21617 mini_ranger_kms.cc:384] Added ranger policy
I20260808 06:58:31.754904 21617 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 06:58:31.754936 21617 mini_ranger_kms.cc:240] Using host: 127.21.28.84
I20260808 06:58:31.759125 21617 mini_ranger_kms.cc:276] Ranger KMS PID: 5763
I20260808 06:58:31.759217 21617 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.28.84:60165
Aug 08, 2026 6:58:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:58:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:58:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 6:58:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 6:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 6:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 06:58:33 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786172313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.28.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:58:34.229714 21617 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.805s user 0.006s sys 0.605s
I20260808 06:58:34.229849 21617 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 06:58:34.229941 21617 mini_ranger_kms.cc:401] 127.21.28.84:60165/kms/v1/keys
Aug 08 06:58:34 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.28.84@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
Aug 08 06:58:34 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.28.84@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
Aug 08 06:58:37 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:58:37.220474 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:33119
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:33119
--ranger_config_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:60165/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:58:37.322042 5848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:58:37.322229 5848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:58:37.322250 5848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:58:37.323319 5848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:58:37.323352 5848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:58:37.323364 5848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:58:37.323375 5848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:58:37.323385 5848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:58:37.325055 5848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.28.84:60165/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:33119
--ranger_config_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:33119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:58:37.325240 5848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:58:37.325403 5848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:58:37.330070 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:58:37.330114 5859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:58:37.330360 5848 server_base.cc:1034] running on GCE node
W20260808 06:58:37.330415 5856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:58:37.330623 5848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:58:37.330865 5848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:58:37.332003 5848 hybrid_clock.cc:648] HybridClock initialized: now 1786172317331995 us; error 33 us; skew 500 ppm
Aug 08 06:58:37 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172317, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:58:37.337738 5848 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:58:37.338155 5848 webserver.cc:460] Webserver started at http://127.21.28.126:38285/ using document root <none> and password file <none>
I20260808 06:58:37.338337 5848 fs_manager.cc:359] Metadata directory not provided
I20260808 06:58:37.338377 5848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:58:37.338501 5848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:58:37 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172317, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:58:37.404737 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "5444f01c392d4d969131728755081554"
format_stamp: "Formatted at 2026-08-08 06:58:37 on dist-test-slave-nj21"
server_key: "d5b3d68024bd0951c4fa99c4d9819b36"
server_key_iv: "47c1fec5a2dea65fc478181d7cbb82b7"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:37.405150 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "5444f01c392d4d969131728755081554"
format_stamp: "Formatted at 2026-08-08 06:58:37 on dist-test-slave-nj21"
server_key: "d5b3d68024bd0951c4fa99c4d9819b36"
server_key_iv: "47c1fec5a2dea65fc478181d7cbb82b7"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:37.406639 5848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 06:58:37.422729 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:37.422916 5848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:58:37.423009 5848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "5444f01c392d4d969131728755081554"
format_stamp: "Formatted at 2026-08-08 06:58:37 on dist-test-slave-nj21"
server_key: "d5b3d68024bd0951c4fa99c4d9819b36"
server_key_iv: "47c1fec5a2dea65fc478181d7cbb82b7"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:37.423102 5848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:58:37.467049 5848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:58:37.469411 5848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:58:37.469672 5848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:58:37.477737 5848 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:33119
I20260808 06:58:37.477769 5948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:33119 every 8 connection(s)
I20260808 06:58:37.478255 5848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:58:37.479017 5950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:58:37.481637 5950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554: Bootstrap starting.
I20260808 06:58:37.482316 5950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554: Neither blocks nor log segments found. Creating new log.
I20260808 06:58:37.482635 5950 log.cc:824] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554: Log is configured to *not* fsync() on all Append() calls
I20260808 06:58:37.483424 5950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554: No bootstrap required, opened a new log
I20260808 06:58:37.486059 5950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5444f01c392d4d969131728755081554" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33119 } }
I20260808 06:58:37.486212 5950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:58:37.486246 5950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5444f01c392d4d969131728755081554, State: Initialized, Role: FOLLOWER
I20260808 06:58:37.486382 5950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5444f01c392d4d969131728755081554" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33119 } }
I20260808 06:58:37.486474 5950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:58:37.486531 5950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:58:37.486585 5950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:58:37.486580 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 5848
I20260808 06:58:37.486665 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:58:37.486867 21617 external_mini_cluster.cc:1428] Setting key ff99fcaa0e97237beed0b3eef3abb11c
I20260808 06:58:37.487361 5950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5444f01c392d4d969131728755081554" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33119 } }
I20260808 06:58:37.487543 5950 leader_election.cc:302] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5444f01c392d4d969131728755081554; no voters:
I20260808 06:58:37.487736 5950 leader_election.cc:288] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:58:37.487772 5957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:58:37.487977 5957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [term 1 LEADER]: Becoming Leader. State: Replica: 5444f01c392d4d969131728755081554, State: Running, Role: LEADER
I20260808 06:58:37.488067 5950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:58:37.488132 5957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5444f01c392d4d969131728755081554" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33119 } }
I20260808 06:58:37.488461 5957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5444f01c392d4d969131728755081554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5444f01c392d4d969131728755081554" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33119 } } }
I20260808 06:58:37.488593 5957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [sys.catalog]: This master's current role is: LEADER
I20260808 06:58:37.488772 5957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5444f01c392d4d969131728755081554. Latest consensus state: current_term: 1 leader_uuid: "5444f01c392d4d969131728755081554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5444f01c392d4d969131728755081554" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33119 } } }
I20260808 06:58:37.488891 5957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554 [sys.catalog]: This master's current role is: LEADER
I20260808 06:58:37.489085 5965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:58:37.489663 5965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:58:37.490511 5950 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 08 06:58:37 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:58:37.499797 5965 catalog_manager.cc:1347] Generated new cluster ID: 7eab774e32584758a75a6630e5af94ab
I20260808 06:58:37.499912 5965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:58:37.504047 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:37.487364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.21.28.126:33119)
0808 06:58:37.487559 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:37.487569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:37.487599 (+ 30us) server_negotiation.cc:407] Connection header received
0808 06:58:37.487734 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:37.487741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:37.487795 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:37.488989 (+ 1194us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:58:37.489092 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:37.490700 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:37.493147 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:37.495284 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:37.498616 (+ 3332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:37.498975 (+ 359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:58:37.498979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:58:37.498979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:58:37.498990 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:58:37.501867 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:37.502310 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:37.502313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:37.502314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:37.502548 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:37.502775 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:37.502776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:37.502776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:37.503079 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:58:37.503607 (+ 528us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:37.503698 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 06:58:37.503813 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260808 06:58:37.512863 5965 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:58:37.513605 5965 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:58:37.520148 5965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554: Generated new TSK 0
I20260808 06:58:37.520375 5965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:58:38.510360 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:60165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33119
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:58:38.654937 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:58:38.655223 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:58:38.655279 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:58:38.656919 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:58:38.656981 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:58:38.657018 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:58:38.659112 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.28.84:60165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:58:38.659373 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:58:38.659643 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:58:38.665230 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:58:38.665472 6039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:58:38.665694 6037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:58:38.666395 6027 server_base.cc:1034] running on GCE node
I20260808 06:58:38.666599 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:58:38.666851 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:58:38.668015 6027 hybrid_clock.cc:648] HybridClock initialized: now 1786172318667688 us; error 339 us; skew 500 ppm
Aug 08 06:58:38 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172318, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:58:38.671224 6027 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:58:38.671734 6027 webserver.cc:460] Webserver started at http://127.21.28.65:33027/ using document root <none> and password file <none>
I20260808 06:58:38.671985 6027 fs_manager.cc:359] Metadata directory not provided
I20260808 06:58:38.672031 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:58:38.672145 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:58:38 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172318, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:58:38.687940 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "8ffec4dce25e4cd1901916abdb615b3b"
format_stamp: "Formatted at 2026-08-08 06:58:38 on dist-test-slave-nj21"
server_key: "087ad028f8b54bddecc0a273913ac684"
server_key_iv: "2b262664c139dd1675dd40afe748fd01"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:38.688504 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "8ffec4dce25e4cd1901916abdb615b3b"
format_stamp: "Formatted at 2026-08-08 06:58:38 on dist-test-slave-nj21"
server_key: "087ad028f8b54bddecc0a273913ac684"
server_key_iv: "2b262664c139dd1675dd40afe748fd01"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:38.690548 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:58:38.706713 6057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:38.706918 6027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260808 06:58:38.707015 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "8ffec4dce25e4cd1901916abdb615b3b"
format_stamp: "Formatted at 2026-08-08 06:58:38 on dist-test-slave-nj21"
server_key: "087ad028f8b54bddecc0a273913ac684"
server_key_iv: "2b262664c139dd1675dd40afe748fd01"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:38.707106 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:58:38.717083 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:58:38.718827 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:58:38.718959 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:58:38.719195 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:58:38.719575 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:58:38.719624 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:38.719653 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:58:38.719717 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:38.731213 6027 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:39117
I20260808 06:58:38.731714 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:58:38.733603 6240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:39117 every 8 connection(s)
I20260808 06:58:38.737092 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 6027
I20260808 06:58:38.737203 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:58:38.737418 21617 external_mini_cluster.cc:1428] Setting key 2250fa02d29f61f7c6ea8859bb10ecae
Aug 08 06:58:38 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172318, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
I20260808 06:58:38.753461 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:38.734269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:47443 (local address 127.21.28.126:33119)
0808 06:58:38.734412 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:38.734414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:38.734625 (+ 211us) server_negotiation.cc:407] Connection header received
0808 06:58:38.735428 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:38.735429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:38.735442 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:38.735510 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:58:38.736279 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:38.736807 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:38.738403 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:38.738665 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:38.738733 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:38.749505 (+ 10772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:58:38.749507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:58:38.749508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:58:38.749517 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:58:38.751654 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:38.752210 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:38.752212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:38.752213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:38.752299 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:38.752625 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:38.752626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:38.752627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:38.752789 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:58:38.752868 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:38.753212 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 06:58:38.753309 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260808 06:58:38.754032 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:38.734240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33119 (local address 127.21.28.65:47443)
0808 06:58:38.734484 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:38.734536 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 06:58:38.735310 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:58:38.735595 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:58:38.735600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:58:38.735957 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:58:38.736184 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:38.736190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:38.736887 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:38.736888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:38.738296 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:38.738299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:38.738743 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:38.738744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:38.738875 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:38.742245 (+ 3370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:58:38.742253 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:58:38.749330 (+ 7077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:58:38.751775 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:58:38.751786 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:58:38.751789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:58:38.752114 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:58:38.752385 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:58:38.752386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:58:38.752386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:58:38.752550 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:58:38.752866 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:58:38.752873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:58:38.753047 (+ 174us) client_negotiation.cc:769] Sending connection context
0808 06:58:38.753571 (+ 524us) client_negotiation.cc:240] Negotiation successful
0808 06:58:38.753750 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":131,"threads_started":1}
I20260808 06:58:38.755324 6243 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33119
I20260808 06:58:38.755445 6243 heartbeater.cc:464] Registering TS with master...
I20260808 06:58:38.755640 6243 heartbeater.cc:511] Master 127.21.28.126:33119 requested a full tablet report, sending...
I20260808 06:58:38.756283 5898 ts_manager.cc:194] Registered new tserver with Master: 8ffec4dce25e4cd1901916abdb615b3b (127.21.28.65:39117)
I20260808 06:58:38.757193 5898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:47443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:58:38.770845 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:60165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33119
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:58:38.909822 6258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:58:38.910090 6258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:58:38.910126 6258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:58:38.911819 6258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:58:38.911880 6258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:58:38.911917 6258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:58:38.914110 6258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.28.84:60165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:58:38.914371 6258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:58:38.914654 6258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:58:38.921705 6269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:58:38.921964 6270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:58:38.922050 6273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:58:38.922318 6258 server_base.cc:1034] running on GCE node
I20260808 06:58:38.922534 6258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:58:38.922780 6258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:58:38.923929 6258 hybrid_clock.cc:648] HybridClock initialized: now 1786172318923908 us; error 33 us; skew 500 ppm
Aug 08 06:58:38 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172318, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:58:38.930994 6258 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:58:38.931543 6258 webserver.cc:460] Webserver started at http://127.21.28.66:41249/ using document root <none> and password file <none>
I20260808 06:58:38.931839 6258 fs_manager.cc:359] Metadata directory not provided
I20260808 06:58:38.931911 6258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:58:38.932040 6258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:58:38 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172318, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:58:38.955714 6258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "ca9fd08f7572442c8969b77e1dcad0f8"
format_stamp: "Formatted at 2026-08-08 06:58:38 on dist-test-slave-nj21"
server_key: "c9807edccfd979e12c1b02946ea302a8"
server_key_iv: "54105d27e71ebd5b9f2cf8fba60d891c"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:38.956264 6258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "ca9fd08f7572442c8969b77e1dcad0f8"
format_stamp: "Formatted at 2026-08-08 06:58:38 on dist-test-slave-nj21"
server_key: "c9807edccfd979e12c1b02946ea302a8"
server_key_iv: "54105d27e71ebd5b9f2cf8fba60d891c"
server_key_version: "kuduclusterkey@0"
Aug 08 06:58:38 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786172318, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:58:38.964252 6258 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.002s
I20260808 06:58:38.995847 6285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:38.996160 6258 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.000s
I20260808 06:58:38.996282 6258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "ca9fd08f7572442c8969b77e1dcad0f8"
format_stamp: "Formatted at 2026-08-08 06:58:38 on dist-test-slave-nj21"
server_key: "c9807edccfd979e12c1b02946ea302a8"
server_key_iv: "54105d27e71ebd5b9f2cf8fba60d891c"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:38.996390 6258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:58:39.021520 6258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:58:39.023885 6258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:58:39.024025 6258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:58:39.024266 6258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:58:39.024715 6258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:58:39.024770 6258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:39.024798 6258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:58:39.024856 6258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:39.039247 6258 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:43585
I20260808 06:58:39.039280 6456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:43585 every 8 connection(s)
I20260808 06:58:39.039819 6258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:58:39.041436 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 6258
I20260808 06:58:39.041535 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:58:39.041743 21617 external_mini_cluster.cc:1428] Setting key e3aa54f6e5f353cb063128be44892882
Aug 08 06:58:39 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172318, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:58:39.050153 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.040631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:46431 (local address 127.21.28.126:33119)
0808 06:58:39.040700 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.040702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.040961 (+ 259us) server_negotiation.cc:407] Connection header received
0808 06:58:39.041576 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.041577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.041594 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.041645 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:58:39.042570 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.043092 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.044318 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.044685 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.044729 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.047455 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:58:39.047456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:58:39.047457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:58:39.047465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:58:39.048784 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:39.049200 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:39.049202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:39.049202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:39.049266 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:39.049513 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:39.049514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:39.049514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:39.049628 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:58:39.049687 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.049953 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.050055 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:58:39.050292 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.040584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33119 (local address 127.21.28.66:46431)
0808 06:58:39.040813 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:39.040855 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:58:39.041463 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:58:39.041714 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:58:39.041718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:58:39.042049 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:58:39.042453 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.042461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.043177 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.043178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.044197 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.044202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.044826 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.044827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.045025 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.045745 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:58:39.045754 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:58:39.047367 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:58:39.048856 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:58:39.048863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:58:39.048865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:58:39.049129 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:58:39.049325 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:58:39.049326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:58:39.049326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:58:39.049454 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:58:39.049685 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:58:39.049691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:58:39.049834 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 06:58:39.049922 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 06:58:39.050069 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260808 06:58:39.050947 6461 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33119
I20260808 06:58:39.051041 6461 heartbeater.cc:464] Registering TS with master...
I20260808 06:58:39.051218 6461 heartbeater.cc:511] Master 127.21.28.126:33119 requested a full tablet report, sending...
I20260808 06:58:39.051594 5898 ts_manager.cc:194] Registered new tserver with Master: ca9fd08f7572442c8969b77e1dcad0f8 (127.21.28.66:43585)
I20260808 06:58:39.051982 5898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:46431
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:58:39.082168 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:60165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33119
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:58:39.236656 6467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:58:39.236977 6467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:58:39.237018 6467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:58:39.238785 6467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:58:39.238847 6467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:58:39.238884 6467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:58:39.241147 6467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.28.84:60165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:58:39.241395 6467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:58:39.241734 6467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:58:39.248093 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:58:39.248171 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:58:39.248366 6480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:58:39.249063 6467 server_base.cc:1034] running on GCE node
I20260808 06:58:39.249279 6467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:58:39.249577 6467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:58:39.250767 6467 hybrid_clock.cc:648] HybridClock initialized: now 1786172319250743 us; error 29 us; skew 500 ppm
Aug 08 06:58:39 dist-test-slave-nj21 krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172319, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:58:39.254482 6467 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:58:39.254984 6467 webserver.cc:460] Webserver started at http://127.21.28.67:46133/ using document root <none> and password file <none>
I20260808 06:58:39.255195 6467 fs_manager.cc:359] Metadata directory not provided
I20260808 06:58:39.255239 6467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:58:39.255359 6467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:58:39 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172319, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:58:39.267429 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "d0659f8b7c314e44aa4a35543e2d37f2"
format_stamp: "Formatted at 2026-08-08 06:58:39 on dist-test-slave-nj21"
server_key: "2a96aa39987ea8dafa92a575e18a6513"
server_key_iv: "136ae5082202ee2e8544b6447b0ce3b0"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:39.268023 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "d0659f8b7c314e44aa4a35543e2d37f2"
format_stamp: "Formatted at 2026-08-08 06:58:39 on dist-test-slave-nj21"
server_key: "2a96aa39987ea8dafa92a575e18a6513"
server_key_iv: "136ae5082202ee2e8544b6447b0ce3b0"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:39.270278 6467 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260808 06:58:39.291586 6494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:39.292225 6467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:58:39.292353 6467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "d0659f8b7c314e44aa4a35543e2d37f2"
format_stamp: "Formatted at 2026-08-08 06:58:39 on dist-test-slave-nj21"
server_key: "2a96aa39987ea8dafa92a575e18a6513"
server_key_iv: "136ae5082202ee2e8544b6447b0ce3b0"
server_key_version: "kuduclusterkey@0"
I20260808 06:58:39.292472 6467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:58:39.320393 6467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:58:39.322157 6467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:58:39.322281 6467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:58:39.322526 6467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:58:39.322840 6467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:58:39.322875 6467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:39.322904 6467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:58:39.322929 6467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:58:39.334245 6467 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:35023
I20260808 06:58:39.334482 6664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:35023 every 8 connection(s)
I20260808 06:58:39.334690 6467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 06:58:39 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172319, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:58:39.344741 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.335610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:58187 (local address 127.21.28.126:33119)
0808 06:58:39.335655 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.335658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.336050 (+ 392us) server_negotiation.cc:407] Connection header received
0808 06:58:39.336698 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.336699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.336713 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.336768 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:58:39.337529 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.338039 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.339070 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.339293 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.339350 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.341987 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:58:39.341989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:58:39.341989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:58:39.341996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:58:39.343223 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:39.343743 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:39.343744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:39.343745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:39.343805 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:39.344075 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:39.344075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:39.344076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:39.344174 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:58:39.344217 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.344559 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.344646 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:58:39.344981 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 6467
I20260808 06:58:39.345003 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.335594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33119 (local address 127.21.28.67:58187)
0808 06:58:39.335905 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:39.335943 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 06:58:39.336596 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:58:39.336844 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:58:39.336848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:58:39.337151 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:58:39.337437 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.337442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.338122 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.338123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.338986 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.338990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.339368 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.339369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.339519 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.340130 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:58:39.340139 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:58:39.341883 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:58:39.343308 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:58:39.343316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:58:39.343319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:58:39.343636 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:58:39.343866 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:58:39.343867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:58:39.343867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:58:39.344013 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:58:39.344227 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:58:39.344233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:58:39.344401 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 06:58:39.344527 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 06:58:39.344715 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":206,"threads_started":1}
I20260808 06:58:39.345069 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:58:39.345288 21617 external_mini_cluster.cc:1428] Setting key 00bc8013b25482f0d0b88f5fcba04f39
I20260808 06:58:39.345585 6667 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33119
I20260808 06:58:39.345686 6667 heartbeater.cc:464] Registering TS with master...
I20260808 06:58:39.345889 6667 heartbeater.cc:511] Master 127.21.28.126:33119 requested a full tablet report, sending...
I20260808 06:58:39.347396 5898 ts_manager.cc:194] Registered new tserver with Master: d0659f8b7c314e44aa4a35543e2d37f2 (127.21.28.67:35023)
I20260808 06:58:39.347832 5898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:58187
I20260808 06:58:39.357025 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:58:39.365636 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.358751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.21.28.126:33119)
0808 06:58:39.358796 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.358799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.359026 (+ 227us) server_negotiation.cc:407] Connection header received
0808 06:58:39.359091 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.359092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.359100 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.359140 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:58:39.359967 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.360441 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.361506 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.361723 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.361782 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.363357 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:58:39.363359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:58:39.363359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:58:39.363365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:58:39.364600 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:39.364986 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:39.364987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:39.364987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:39.365030 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:58:39.365211 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:58:39.365211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:58:39.365211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:58:39.365300 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:58:39.365364 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.365521 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.365574 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:58:39.367271 5898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:58:39.369097 5898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:58:39.388875 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.380002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:39117 (local address 127.0.0.1:44104)
0808 06:58:39.381167 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:39.381214 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 06:58:39.381293 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:58:39.384092 (+ 2799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:58:39.384096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:58:39.384098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:58:39.384182 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.384184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.385697 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.385698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.387134 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.387137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.388557 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.388558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.388648 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.388668 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 06:58:39.388723 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 06:58:39.388782 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":1058,"threads_started":1}
I20260808 06:58:39.389415 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.380280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.21.28.65:39117)
0808 06:58:39.383806 (+ 3526us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.383811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.383823 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:58:39.383862 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.383864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.383883 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.383974 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:58:39.384281 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.385582 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.387250 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.388448 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.389091 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.389169 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.389200 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.389288 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3486,"thread_start_us":63,"threads_started":1}
I20260808 06:58:39.390213 6131 tablet_service.cc:1467] Processing CreateTablet for tablet c51a4f37a18e4fe38c3649609252920a (DEFAULT_TABLE table=test-table [id=7aa90385481f4bc1afefe099da6d7600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:58:39.390681 6131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51a4f37a18e4fe38c3649609252920a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:58:39.390894 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.380647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.21.28.67:35023)
0808 06:58:39.380975 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.380981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.384977 (+ 3996us) server_negotiation.cc:407] Connection header received
0808 06:58:39.385131 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.385134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.385152 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.385210 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:58:39.385472 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.386771 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.388327 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.389984 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.390039 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.390160 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.390631 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.390677 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":152,"threads_started":1}
I20260808 06:58:39.391301 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.380205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35023 (local address 127.0.0.1:45770)
0808 06:58:39.384907 (+ 4702us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:39.384916 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:58:39.385007 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:58:39.385327 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:58:39.385328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:58:39.385330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:58:39.385380 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.385382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.386890 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.386891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.388079 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.388082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.390385 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.390386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.390523 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.390528 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:58:39.391172 (+ 644us) client_negotiation.cc:240] Negotiation successful
0808 06:58:39.391224 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4669,"thread_start_us":150,"threads_started":1}
I20260808 06:58:39.392292 6566 tablet_service.cc:1467] Processing CreateTablet for tablet c51a4f37a18e4fe38c3649609252920a (DEFAULT_TABLE table=test-table [id=7aa90385481f4bc1afefe099da6d7600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:58:39.392709 6566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51a4f37a18e4fe38c3649609252920a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:58:39.395866 6697 tablet_bootstrap.cc:492] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b: Bootstrap starting.
I20260808 06:58:39.397676 6697 tablet_bootstrap.cc:654] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b: Neither blocks nor log segments found. Creating new log.
I20260808 06:58:39.398578 6697 log.cc:824] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:58:39.399091 6698 tablet_bootstrap.cc:492] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Bootstrap starting.
I20260808 06:58:39.399693 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.380006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:43585 (local address 127.0.0.1:55576)
0808 06:58:39.384703 (+ 4697us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:39.384716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:58:39.384775 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:58:39.390206 (+ 5431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:58:39.390207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:58:39.390209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:58:39.390253 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.390255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.391802 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.391803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.392887 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.392890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.399398 (+ 6508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.399400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.399482 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.399487 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:58:39.399542 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 06:58:39.399591 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4648,"thread_start_us":967,"threads_started":1}
I20260808 06:58:39.399821 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.380460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.21.28.66:43585)
0808 06:58:39.389862 (+ 9402us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.389866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.389878 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:58:39.389948 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.389950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.389968 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.390133 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:58:39.390328 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.391608 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.398308 (+ 6700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.399312 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.399406 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.399596 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.399630 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.399696 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9331,"thread_start_us":138,"threads_started":1}
I20260808 06:58:39.400590 6357 tablet_service.cc:1467] Processing CreateTablet for tablet c51a4f37a18e4fe38c3649609252920a (DEFAULT_TABLE table=test-table [id=7aa90385481f4bc1afefe099da6d7600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:58:39.401010 6357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51a4f37a18e4fe38c3649609252920a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:58:39.407536 6698 tablet_bootstrap.cc:654] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Neither blocks nor log segments found. Creating new log.
I20260808 06:58:39.408108 6697 tablet_bootstrap.cc:492] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b: No bootstrap required, opened a new log
I20260808 06:58:39.408183 6697 ts_tablet_manager.cc:1397] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260808 06:58:39.413050 6697 raft_consensus.cc:348] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.413365 6697 raft_consensus.cc:374] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:58:39.413458 6697 raft_consensus.cc:729] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffec4dce25e4cd1901916abdb615b3b, State: Initialized, Role: FOLLOWER
I20260808 06:58:39.413681 6697 consensus_queue.cc:260] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.414089 6697 ts_tablet_manager.cc:1428] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 06:58:39.414155 6243 heartbeater.cc:503] Master 127.21.28.126:33119 was elected leader, sending a full tablet report...
I20260808 06:58:39.415043 6698 log.cc:824] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Log is configured to *not* fsync() on all Append() calls
I20260808 06:58:39.415629 6703 tablet_bootstrap.cc:492] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Bootstrap starting.
I20260808 06:58:39.416358 6698 tablet_bootstrap.cc:492] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: No bootstrap required, opened a new log
I20260808 06:58:39.416482 6698 ts_tablet_manager.cc:1397] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20260808 06:58:39.417847 6703 tablet_bootstrap.cc:654] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Neither blocks nor log segments found. Creating new log.
I20260808 06:58:39.418615 6703 log.cc:824] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Log is configured to *not* fsync() on all Append() calls
I20260808 06:58:39.420425 6698 raft_consensus.cc:348] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.420634 6698 raft_consensus.cc:374] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:58:39.420713 6698 raft_consensus.cc:729] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0659f8b7c314e44aa4a35543e2d37f2, State: Initialized, Role: FOLLOWER
I20260808 06:58:39.420872 6698 consensus_queue.cc:260] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.421219 6698 ts_tablet_manager.cc:1428] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 06:58:39.421389 6667 heartbeater.cc:503] Master 127.21.28.126:33119 was elected leader, sending a full tablet report...
I20260808 06:58:39.422477 6703 tablet_bootstrap.cc:492] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: No bootstrap required, opened a new log
I20260808 06:58:39.422593 6703 ts_tablet_manager.cc:1397] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260808 06:58:39.425592 6703 raft_consensus.cc:348] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.425716 6703 raft_consensus.cc:374] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:58:39.425746 6703 raft_consensus.cc:729] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9fd08f7572442c8969b77e1dcad0f8, State: Initialized, Role: FOLLOWER
I20260808 06:58:39.425896 6703 consensus_queue.cc:260] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.426194 6703 ts_tablet_manager.cc:1428] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260808 06:58:39.426291 6461 heartbeater.cc:503] Master 127.21.28.126:33119 was elected leader, sending a full tablet report...
W20260808 06:58:39.484678 6244 tablet.cc:2369] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:58:39.540839 6462 tablet.cc:2369] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 06:58:39 dist-test-slave-nj21 krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172318, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
W20260808 06:58:39.585480 6668 tablet.cc:2369] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:58:39.591845 6708 raft_consensus.cc:482] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:58:39.592000 6708 raft_consensus.cc:504] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.592450 6708 leader_election.cc:288] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ffec4dce25e4cd1901916abdb615b3b (127.21.28.65:39117), d0659f8b7c314e44aa4a35543e2d37f2 (127.21.28.67:35023)
I20260808 06:58:39.604676 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.593262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:34201 (local address 127.21.28.67:35023)
0808 06:58:39.593312 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.593314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.600367 (+ 7053us) server_negotiation.cc:407] Connection header received
0808 06:58:39.600472 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.600473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.600483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.600512 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:58:39.600817 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.601876 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.603660 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.604221 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.604276 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.604286 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.604551 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.604598 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 06:58:39.605304 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.592866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35023 (local address 127.21.28.66:34201)
0808 06:58:39.600283 (+ 7417us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:39.600298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:58:39.600399 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:58:39.600587 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:58:39.600588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:58:39.600590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:58:39.600720 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.600724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.602062 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.602063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.603449 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.603452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.604340 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.604341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.604455 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.604460 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:58:39.605113 (+ 653us) client_negotiation.cc:240] Negotiation successful
0808 06:58:39.605163 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7397,"thread_start_us":57,"threads_started":1}
I20260808 06:58:39.605454 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.593111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:39479 (local address 127.21.28.65:39117)
0808 06:58:39.593162 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.593164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.593174 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:58:39.593207 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.593208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.593221 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.593282 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:58:39.601982 (+ 8700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.602712 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.604286 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.605025 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.605075 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.605084 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.605345 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.605391 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:58:39.605777 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.592693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:39117 (local address 127.21.28.66:39479)
0808 06:58:39.592964 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:39.592999 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 06:58:39.593054 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:58:39.601651 (+ 8597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:58:39.601652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:58:39.601655 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:58:39.601858 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.601860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.602825 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.602826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.604182 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:58:39.604185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.605139 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.605140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:58:39.605250 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.605255 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:58:39.605526 (+ 271us) client_negotiation.cc:240] Negotiation successful
0808 06:58:39.605569 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":97,"threads_started":1}
I20260808 06:58:39.605939 6581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51a4f37a18e4fe38c3649609252920a" candidate_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" is_pre_election: true
I20260808 06:58:39.606159 6581 raft_consensus.cc:2454] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9fd08f7572442c8969b77e1dcad0f8 in term 0.
I20260808 06:58:39.606153 6164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51a4f37a18e4fe38c3649609252920a" candidate_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffec4dce25e4cd1901916abdb615b3b" is_pre_election: true
I20260808 06:58:39.606276 6164 raft_consensus.cc:2454] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9fd08f7572442c8969b77e1dcad0f8 in term 0.
I20260808 06:58:39.606483 6288 leader_election.cc:302] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca9fd08f7572442c8969b77e1dcad0f8, d0659f8b7c314e44aa4a35543e2d37f2; no voters:
I20260808 06:58:39.606617 6708 raft_consensus.cc:2781] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:58:39.606668 6708 raft_consensus.cc:482] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:58:39.606691 6708 raft_consensus.cc:3037] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:58:39.607784 6708 raft_consensus.cc:504] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.607914 6708 leader_election.cc:288] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8ffec4dce25e4cd1901916abdb615b3b (127.21.28.65:39117), d0659f8b7c314e44aa4a35543e2d37f2 (127.21.28.67:35023)
I20260808 06:58:39.608331 6164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51a4f37a18e4fe38c3649609252920a" candidate_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffec4dce25e4cd1901916abdb615b3b"
I20260808 06:58:39.608428 6164 raft_consensus.cc:3037] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:58:39.610049 6581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51a4f37a18e4fe38c3649609252920a" candidate_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0659f8b7c314e44aa4a35543e2d37f2"
I20260808 06:58:39.610126 6581 raft_consensus.cc:3037] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:58:39.610271 6164 raft_consensus.cc:2454] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9fd08f7572442c8969b77e1dcad0f8 in term 1.
I20260808 06:58:39.610456 6287 leader_election.cc:302] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ffec4dce25e4cd1901916abdb615b3b, ca9fd08f7572442c8969b77e1dcad0f8; no voters:
I20260808 06:58:39.611183 6581 raft_consensus.cc:2454] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9fd08f7572442c8969b77e1dcad0f8 in term 1.
I20260808 06:58:39.611366 6708 raft_consensus.cc:2781] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:58:39.611548 6708 raft_consensus.cc:686] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 1 LEADER]: Becoming Leader. State: Replica: ca9fd08f7572442c8969b77e1dcad0f8, State: Running, Role: LEADER
I20260808 06:58:39.611652 6708 consensus_queue.cc:237] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } }
I20260808 06:58:39.612468 5898 catalog_manager.cc:5495] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca9fd08f7572442c8969b77e1dcad0f8 (127.21.28.66). New cstate: current_term: 1 leader_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca9fd08f7572442c8969b77e1dcad0f8" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 } health_report { overall_health: UNKNOWN } } }
I20260808 06:58:39.671413 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.668171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.21.28.66:43585)
0808 06:58:39.668247 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 06:58:39.668250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:58:39.668260 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:58:39.668305 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:58:39.668306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:58:39.668331 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:58:39.668410 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:58:39.668581 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.669333 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.670489 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:58:39.670781 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:58:39.670834 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:58:39.671001 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:58:39.671103 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:58:39.671157 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:58:39.671220 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 06:58:39.671241 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 06:58:39.673419 6164 raft_consensus.cc:1261] T c51a4f37a18e4fe38c3649609252920a P 8ffec4dce25e4cd1901916abdb615b3b [term 1 FOLLOWER]: Refusing update from remote peer ca9fd08f7572442c8969b77e1dcad0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:58:39.673815 6581 raft_consensus.cc:1261] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 1 FOLLOWER]: Refusing update from remote peer ca9fd08f7572442c8969b77e1dcad0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:58:39.673769 6714 consensus_queue.cc:1035] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ffec4dce25e4cd1901916abdb615b3b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 39117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:58:39.674099 6714 consensus_queue.cc:1035] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0659f8b7c314e44aa4a35543e2d37f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:58:39.676986 6719 mvcc.cc:205] Tried to move back new op lower bound from 7316161821379391488 to 7316161821129957376. Current Snapshot: MvccSnapshot[applied={T|T < 7316161821379391488}]
I20260808 06:58:39.677053 6720 mvcc.cc:205] Tried to move back new op lower bound from 7316161821379391488 to 7316161821129957376. Current Snapshot: MvccSnapshot[applied={T|T < 7316161821379391488}]
I20260808 06:58:39.692691 6721 mvcc.cc:205] Tried to move back new op lower bound from 7316161821379391488 to 7316161821129957376. Current Snapshot: MvccSnapshot[applied={T|T < 7316161821379391488}]
I20260808 06:58:39.694307 5897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:58:39.694420 5897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:58:39.695355 5897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5444f01c392d4d969131728755081554: Sending DeleteTablet for 3 replicas of tablet c51a4f37a18e4fe38c3649609252920a
I20260808 06:58:39.695808 6566 tablet_service.cc:1514] Processing DeleteTablet for tablet c51a4f37a18e4fe38c3649609252920a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:58:39 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:45770
I20260808 06:58:39.695815 6131 tablet_service.cc:1514] Processing DeleteTablet for tablet c51a4f37a18e4fe38c3649609252920a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:58:39 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:44104
I20260808 06:58:39.696205 6733 tablet_replica.cc:330] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: stopping tablet replica
I20260808 06:58:39.696354 6733 raft_consensus.cc:2229] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:58:39.696524 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 6027
I20260808 06:58:39.696573 6733 raft_consensus.cc:2258] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:58:39.701308 6733 ts_tablet_manager.cc:1905] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:58:39.703438 6733 ts_tablet_manager.cc:1918] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:58:39.703508 6733 log.cc:1196] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/c51a4f37a18e4fe38c3649609252920a
I20260808 06:58:39.704170 6733 ts_tablet_manager.cc:1939] T c51a4f37a18e4fe38c3649609252920a P d0659f8b7c314e44aa4a35543e2d37f2: Deleting consensus metadata
I20260808 06:58:39.704609 5880 catalog_manager.cc:4841] TS d0659f8b7c314e44aa4a35543e2d37f2 (127.21.28.67:35023): tablet c51a4f37a18e4fe38c3649609252920a (table test-table [id=7aa90385481f4bc1afefe099da6d7600]) successfully deleted
I20260808 06:58:39.708593 6357 tablet_service.cc:1514] Processing DeleteTablet for tablet c51a4f37a18e4fe38c3649609252920a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:58:39 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:55576
I20260808 06:58:39.709513 6737 tablet_replica.cc:330] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: stopping tablet replica
I20260808 06:58:39.709676 6737 raft_consensus.cc:2229] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:58:39.709873 6737 raft_consensus.cc:2258] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:58:39.710458 6737 ts_tablet_manager.cc:1905] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:58:39.712709 6737 ts_tablet_manager.cc:1918] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:58:39.712849 6737 log.cc:1196] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/c51a4f37a18e4fe38c3649609252920a
I20260808 06:58:39.713183 6737 ts_tablet_manager.cc:1939] T c51a4f37a18e4fe38c3649609252920a P ca9fd08f7572442c8969b77e1dcad0f8: Deleting consensus metadata
I20260808 06:58:39.713721 5878 catalog_manager.cc:4841] TS ca9fd08f7572442c8969b77e1dcad0f8 (127.21.28.66:43585): tablet c51a4f37a18e4fe38c3649609252920a (table test-table [id=7aa90385481f4bc1afefe099da6d7600]) successfully deleted
I20260808 06:58:39.716182 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 6258
W20260808 06:58:39.719802 5879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:39117 (error 108)
I20260808 06:58:39.727169 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 6467
I20260808 06:58:39.727331 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:58:39.727188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:39117 (local address 127.0.0.1:44106)
0808 06:58:39.727255 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 06:58:39.727295 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:39117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 06:58:39.733012 5879 catalog_manager.cc:4584] TS 8ffec4dce25e4cd1901916abdb615b3b (127.21.28.65:39117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c51a4f37a18e4fe38c3649609252920a: Network error: Client connection negotiation failed: client connection to 127.21.28.65:39117: connect: Connection refused (error 111)
I20260808 06:58:39.736531 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 5848
I20260808 06:58:39.747402 21617 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 06:58:40.090665 21617 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 06:58:40.090766 21617 mini_ranger.cc:66] Stopping Ranger...
I20260808 06:58:40.434428 21617 mini_ranger.cc:68] Stopped Ranger
2026-08-08 06:58:40.434 UTC [5176] LOG: received smart shutdown request
2026-08-08 06:58:40.438 UTC [5176] LOG: background worker "logical replication launcher" (PID 5194) exited with exit code 1
2026-08-08 06:58:40.438 UTC [5185] LOG: shutting down
2026-08-08 06:58:40.568 UTC [5176] LOG: database system is shut down
2026-08-08T06:58:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46699 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:58:40 dist-test-slave-nj21 krb5kdc[6756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:58:40 dist-test-slave-nj21 krb5kdc[6756](info): set up 2 sockets
Aug 08 06:58:40 dist-test-slave-nj21 krb5kdc[6756](info): commencing operation
krb5kdc: starting...
W20260808 06:58:41.790314 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:58:41 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:58:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.28.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.28.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/rangeradmin_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/rangeradmin_127.21.28.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.28.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.28.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/rangerlookup_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/rangerlookup_127.21.28.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.28.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.28.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/HTTP_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/HTTP_127.21.28.84@KRBTEST.COM.keytab.
I20260808 06:58:41.939213 21617 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDMxMqG/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/postgres -l logfile start
2026-08-08 06:58:44.247 UTC [6805] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 06:58:44.247 UTC [6805] LOG: listening on IPv4 address "127.21.28.84", port 40793
2026-08-08 06:58:44.250 UTC [6805] LOG: listening on Unix socket "/tmp/.s.PGSQL.40793"
2026-08-08 06:58:44.279 UTC [6810] LOG: database system was shut down at 2026-08-08 06:58:42 UTC
2026-08-08 06:58:44.283 UTC [6805] LOG: database system is ready to accept connections
I20260808 06:58:45.322230 21617 mini_postgres.cc:96] Postgres bound to 40793
2026-08-08 06:58:45.328 UTC [6825] FATAL: database "slave" does not exist
127.21.28.84:40793 - accepting connections
I20260808 06:58:45.329169 21617 mini_ranger.cc:161] Starting Ranger...
I20260808 06:58:45.343031 21617 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 06:58:45.752336 21617 mini_ranger.cc:87] Created ranger Postgres database
I20260808 06:58:45.752401 21617 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-admin
2026-08-08 06:58:46,020 [I] DB FLAVOR :POSTGRES
2026-08-08 06:58:46,021 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 06:58:46,021 [I] Checking connection..
2026-08-08 06:58:46,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 06:58:46,280 [I] Checking connection passed.
2026-08-08 06:58:46,280 [I] --------- Verifying version history table ---------
2026-08-08 06:58:46,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:58:46,534 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 06:58:46,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:58:46,789 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 06:58:46,789 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 06:58:46,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 06:58:47.042 UTC [6934] WARNING: there is no transaction in progress
2026-08-08 06:58:47,050 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 06:58:47,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:58:47,292 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 06:58:47,292 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 06:58:47,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 06:58:47,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 06:58:47,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nj21.c.gcp-upstream.internal','N') ;"
2026-08-08 06:58:47,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 06:58:48,246 [I] Table x_portal_user does not exist in database ranger
2026-08-08 06:58:48,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 06:58:48,499 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 06:58:48,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 06:58:48,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 06:58:48,996 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 06:58:48,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 06:58:49.883 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:49.897 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:49.911 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:49.924 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:49.938 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:49.954 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.003 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.009 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.023 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.032 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.042 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.055 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.064 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.071 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.077 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50.083 UTC [7100] WARNING: there is no transaction in progress
2026-08-08 06:58:50,623 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 06:58:50,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 06:58:50,857 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 06:58:50,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 06:58:51,118 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 06:58:51,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 06:58:51,378 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 06:58:51,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 06:58:51,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 06:58:51,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nj21.c.gcp-upstream.internal';"
2026-08-08 06:58:52,108 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 06:58:52,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 06:58:52,363 [I] DB_PATCHES have already been applied
I20260808 06:58:52.368713 21617 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDMxMqG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 06:58:52.368778 21617 mini_ranger.cc:193] Using host: 127.21.28.84
I20260808 06:58:52.372399 21617 mini_ranger.cc:227] Ranger admin URL: http://127.21.28.84:52703
Aug 08, 2026 6:58:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:58:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:58:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 6:58:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 6:58:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 6:58:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 6:58:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 06:58:53.659945 21617 mini_ranger.cc:160] Time spent starting Ranger: real 8.331s user 0.000s sys 0.009s
Aug 08, 2026 6:59:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 6:59:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 6:59:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 6:59:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 6:59:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 06:59:14.804353 21617 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.28.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.28.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/rangerkms_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.28.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/rangerkms_127.21.28.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.28.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.28.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.28.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/HTTP_127.21.28.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.28.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/HTTP_127.21.28.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 06:59:14 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172354, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 06:59:14.882653 21617 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 06:59:14.892366 21617 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 06:59:15.600399 21617 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 06:59:15.600471 21617 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms
I20260808 06:59:15.600483 21617 mini_ranger_kms.cc:227] Using postgres at 127.21.28.84:40793
2026-08-08 06:59:15,784 [I] DB FLAVOR :POSTGRES
2026-08-08 06:59:15,784 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 06:59:15,784 [I] Checking connection
2026-08-08 06:59:15,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 06:59:16,016 [I] connection success
2026-08-08 06:59:16,016 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 06:59:16,016 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 06:59:16,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 06:59:16,253 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 06:59:16,254 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 06:59:16,254 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 06:59:16,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.28.84:40793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 06:59:16,498 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 06:59:16.949327 21617 mini_ranger_kms.cc:310] Created kms service
I20260808 06:59:17.163062 21617 mini_ranger_kms.cc:326] Created kudu user
I20260808 06:59:17.209616 21617 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 06:59:17 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172354, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:17.618158 21617 mini_ranger_kms.cc:384] Added ranger policy
I20260808 06:59:17.618273 21617 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDMxMqG/build/release/bin/postgresql.jar:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDMxMqG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 06:59:17.618291 21617 mini_ranger_kms.cc:240] Using host: 127.21.28.84
I20260808 06:59:17.622064 21617 mini_ranger_kms.cc:276] Ranger KMS PID: 7386
I20260808 06:59:17.622134 21617 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.28.84:53843
Aug 08, 2026 6:59:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:59:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:59:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 6:59:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 6:59:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 6:59:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 06:59:20 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786172360, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.28.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:20.141832 21617 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.259s user 0.008s sys 0.159s
I20260808 06:59:20.141932 21617 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 06:59:20.141981 21617 mini_ranger_kms.cc:401] 127.21.28.84:53843/kms/v1/keys
Aug 08 06:59:20 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172360, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.28.84@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
Aug 08 06:59:20 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172360, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.28.84@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
Aug 08 06:59:23 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:59:23.188710 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46877
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46877
--ranger_config_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:53843/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:23.272514 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:23.272709 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:23.272738 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:23.273873 7466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:23.273922 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:23.273941 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:23.273957 7466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:23.273970 7466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:23.275507 7466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.28.84:53843/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46877
--ranger_config_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:23.275761 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:23.275967 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:23.280675 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:23.280735 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:23.281117 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:23.281615 7466 server_base.cc:1034] running on GCE node
I20260808 06:59:23.281821 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:23.282094 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:23.283241 7466 hybrid_clock.cc:648] HybridClock initialized: now 1786172363283229 us; error 31 us; skew 500 ppm
Aug 08 06:59:23 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:23.285843 7466 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:59:23.286291 7466 webserver.cc:460] Webserver started at http://127.21.28.126:40467/ using document root <none> and password file <none>
I20260808 06:59:23.286499 7466 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:23.286538 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:23.286660 7466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:59:23 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172363, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:23.343793 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "6f488023438945048f59bf8df56a0aa2"
format_stamp: "Formatted at 2026-08-08 06:59:23 on dist-test-slave-nj21"
server_key: "67af5269c1d89b5f122df8cd1932f6bc"
server_key_iv: "6b0d1c8c01f8878a182d665e930b38c6"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:23.344200 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "6f488023438945048f59bf8df56a0aa2"
format_stamp: "Formatted at 2026-08-08 06:59:23 on dist-test-slave-nj21"
server_key: "67af5269c1d89b5f122df8cd1932f6bc"
server_key_iv: "6b0d1c8c01f8878a182d665e930b38c6"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:23.345733 7466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:59:23.362035 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:23.362368 7466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:23.362524 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "6f488023438945048f59bf8df56a0aa2"
format_stamp: "Formatted at 2026-08-08 06:59:23 on dist-test-slave-nj21"
server_key: "67af5269c1d89b5f122df8cd1932f6bc"
server_key_iv: "6b0d1c8c01f8878a182d665e930b38c6"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:23.362622 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:23.386507 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:23.388269 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:23.388408 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:23.393373 7466 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46877
I20260808 06:59:23.393422 7580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46877 every 8 connection(s)
I20260808 06:59:23.393786 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:59:23.394277 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 7466
I20260808 06:59:23.394379 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:59:23.394595 21617 external_mini_cluster.cc:1428] Setting key 4d857843ebf2b1753807d2e73318dc96
I20260808 06:59:23.394646 7581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:23.398453 7581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Bootstrap starting.
I20260808 06:59:23.399166 7581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:23.399525 7581 log.cc:824] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:23.400220 7581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: No bootstrap required, opened a new log
Aug 08 06:59:23 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:23.402582 7581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } }
I20260808 06:59:23.402747 7581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:23.402818 7581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f488023438945048f59bf8df56a0aa2, State: Initialized, Role: FOLLOWER
I20260808 06:59:23.402956 7581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } }
I20260808 06:59:23.403028 7581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:59:23.403060 7581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:59:23.403119 7581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:23.403757 7581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } }
I20260808 06:59:23.403863 7581 leader_election.cc:302] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f488023438945048f59bf8df56a0aa2; no voters:
I20260808 06:59:23.404034 7581 leader_election.cc:288] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:59:23.404304 7581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:23.404282 7588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:23.404546 7588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f488023438945048f59bf8df56a0aa2, State: Running, Role: LEADER
I20260808 06:59:23.405380 7588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } }
I20260808 06:59:23.406055 7589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f488023438945048f59bf8df56a0aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } } }
I20260808 06:59:23.406127 7581 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 06:59:23.406147 7589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:23.406569 7611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:59:23.409371 7591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f488023438945048f59bf8df56a0aa2. Latest consensus state: current_term: 1 leader_uuid: "6f488023438945048f59bf8df56a0aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } } }
I20260808 06:59:23.409632 7591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:23.410274 7611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:59:23.410835 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:23.395133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.21.28.126:46877)
0808 06:59:23.395349 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:23.395358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:23.395390 (+ 32us) server_negotiation.cc:407] Connection header received
0808 06:59:23.396313 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:23.396323 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:23.396387 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:23.396532 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:23.397291 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:23.398686 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:23.399750 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:23.400340 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:23.400438 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:23.403104 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:23.403108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:23.403108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:23.403120 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:23.405153 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:23.405550 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:23.405554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:23.405554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:23.405673 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:23.406477 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:23.406478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:23.406478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:23.406770 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:23.409691 (+ 2921us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:23.409907 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 06:59:23.410513 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":109,"threads_started":1}
I20260808 06:59:23.412763 7611 catalog_manager.cc:1347] Generated new cluster ID: b07db89639594294ae1cb6b12ac8abf1
I20260808 06:59:23.412814 7611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:59:23.429924 7611 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:59:23.430713 7611 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:59:23.442133 7611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Generated new TSK 0
I20260808 06:59:23.442404 7611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:59:24.175657 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:53843/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46877
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:24.265671 7656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:24.265859 7656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:24.265892 7656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:24.267035 7656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:24.267074 7656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:24.267109 7656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:59:24.268587 7656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.28.84:53843/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:24.268805 7656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:24.269030 7656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:24.273910 7666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:24.273954 7663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:24.274140 7664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:24.274338 7656 server_base.cc:1034] running on GCE node
I20260808 06:59:24.274526 7656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:24.274729 7656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:24.275857 7656 hybrid_clock.cc:648] HybridClock initialized: now 1786172364275845 us; error 35 us; skew 500 ppm
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:24.278586 7656 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:59:24.279106 7656 webserver.cc:460] Webserver started at http://127.21.28.65:45095/ using document root <none> and password file <none>
I20260808 06:59:24.279400 7656 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:24.279472 7656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:24.279618 7656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:24.294171 7656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "3ceab25a81d54e879dd9cb5a25374661"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "08b2560b4869510afb694232cd26f471"
server_key_iv: "3c567279a4d805ba2531a90cc091684a"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.294669 7656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "3ceab25a81d54e879dd9cb5a25374661"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "08b2560b4869510afb694232cd26f471"
server_key_iv: "3c567279a4d805ba2531a90cc091684a"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.296106 7656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:59:24.308415 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.308583 7656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:59:24.308669 7656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "3ceab25a81d54e879dd9cb5a25374661"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "08b2560b4869510afb694232cd26f471"
server_key_iv: "3c567279a4d805ba2531a90cc091684a"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.308744 7656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:24.339004 7656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:24.341540 7656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:24.341699 7656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:24.341962 7656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:24.342367 7656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:24.342417 7656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.342485 7656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:24.342516 7656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.350291 7656 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:34433
I20260808 06:59:24.350471 7853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:34433 every 8 connection(s)
I20260808 06:59:24.350754 7656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:59:24.354243 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 7656
I20260808 06:59:24.354358 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:59:24.354615 21617 external_mini_cluster.cc:1428] Setting key 22987c2162437b20d1436818e70cde5b
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:24.362761 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:24.351570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:43511 (local address 127.21.28.126:46877)
0808 06:59:24.351756 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:24.351759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:24.351867 (+ 108us) server_negotiation.cc:407] Connection header received
0808 06:59:24.352657 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:24.352658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:24.352673 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:24.352758 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:24.353497 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.354023 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.355174 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.355503 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.355593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:24.358160 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:24.358162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:24.358163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:24.358172 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:24.359930 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:24.361624 (+ 1694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:24.361626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:24.361626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:24.361700 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:24.362004 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:24.362005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:24.362005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:24.362151 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:24.362219 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:24.362565 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 06:59:24.362654 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
I20260808 06:59:24.364121 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:24.351517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46877 (local address 127.21.28.65:43511)
0808 06:59:24.351734 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:24.351785 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:24.352542 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:24.352842 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:24.352847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:24.353108 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:24.353404 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:24.353410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.354105 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.354106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:24.355093 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:24.355096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.355621 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.355622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:24.355788 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:24.356359 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:24.356369 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:24.358050 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:24.361189 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:24.361208 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:24.361210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:24.361520 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:24.361776 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:24.361777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:24.361778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:24.361923 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:24.362231 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:24.362238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:24.362403 (+ 165us) client_negotiation.cc:769] Sending connection context
0808 06:59:24.362534 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 06:59:24.363877 (+ 1343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":104,"threads_started":1}
I20260808 06:59:24.364722 7854 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46877
I20260808 06:59:24.364825 7854 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:24.365005 7854 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:24.365542 7517 ts_manager.cc:194] Registered new tserver with Master: 3ceab25a81d54e879dd9cb5a25374661 (127.21.28.65:34433)
I20260808 06:59:24.366155 7517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:43511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:59:24.382786 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:53843/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46877
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:24.499111 7865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:24.499385 7865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:24.499423 7865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:24.501099 7865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:24.501154 7865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:24.501192 7865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:59:24.503257 7865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.28.84:53843/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:24.503489 7865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:24.503810 7865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:24.510210 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:24.510234 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:24.510483 7880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:24.510718 7865 server_base.cc:1034] running on GCE node
I20260808 06:59:24.510926 7865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:24.511201 7865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:24.512370 7865 hybrid_clock.cc:648] HybridClock initialized: now 1786172364512349 us; error 33 us; skew 500 ppm
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:24.515623 7865 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:59:24.516180 7865 webserver.cc:460] Webserver started at http://127.21.28.66:38835/ using document root <none> and password file <none>
I20260808 06:59:24.516439 7865 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:24.516503 7865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:24.516633 7865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:24.533340 7865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "313098fe2c9c422b87aede7e5e87c8db"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "28624605baeb8c93a2efaa71f498af2c"
server_key_iv: "17ed3054ddedcfccb857509e647fa2c8"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.533811 7865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "313098fe2c9c422b87aede7e5e87c8db"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "28624605baeb8c93a2efaa71f498af2c"
server_key_iv: "17ed3054ddedcfccb857509e647fa2c8"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.535614 7865 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:24.568583 7894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.569224 7865 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260808 06:59:24.569357 7865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "313098fe2c9c422b87aede7e5e87c8db"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "28624605baeb8c93a2efaa71f498af2c"
server_key_iv: "17ed3054ddedcfccb857509e647fa2c8"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.569515 7865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:24.596668 7865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:24.598945 7865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:24.599086 7865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:24.599299 7865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:24.599709 7865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:24.599754 7865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.599792 7865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:24.599817 7865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.607947 7865 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:37569
I20260808 06:59:24.608033 8079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:37569 every 8 connection(s)
I20260808 06:59:24.608353 7865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:24.616505 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 7865
I20260808 06:59:24.616605 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:59:24.616833 21617 external_mini_cluster.cc:1428] Setting key 02486c2f90c1a6b988c5805bdeb28506
I20260808 06:59:24.618235 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:24.609106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:54749 (local address 127.21.28.126:46877)
0808 06:59:24.609184 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:24.609186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:24.609401 (+ 215us) server_negotiation.cc:407] Connection header received
0808 06:59:24.610132 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:24.610133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:24.610144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:24.610187 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:24.610959 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.611463 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.612963 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.613162 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.613213 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:24.615459 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:24.615460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:24.615461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:24.615469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:24.616927 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:24.617325 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:24.617326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:24.617327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:24.617387 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:24.617609 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:24.617611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:24.617611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:24.617726 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:24.617793 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:24.618038 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 06:59:24.618132 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:59:24.618378 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:24.609049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46877 (local address 127.21.28.66:54749)
0808 06:59:24.609258 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:24.609306 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:24.610033 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:24.610260 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:24.610265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:24.610541 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:24.610860 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:24.610869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.611560 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.611561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:24.612880 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:24.612884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.613259 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.613261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:24.613416 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:24.613902 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:24.613910 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:24.615375 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:24.617019 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:24.617027 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:24.617029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:24.617247 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:24.617450 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:24.617451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:24.617451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:24.617554 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:24.617782 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:24.617789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:24.617923 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 06:59:24.618010 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 06:59:24.618145 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":112,"threads_started":1}
I20260808 06:59:24.619046 8080 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46877
I20260808 06:59:24.619149 8080 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:24.619329 8080 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:24.619779 7517 ts_manager.cc:194] Registered new tserver with Master: 313098fe2c9c422b87aede7e5e87c8db (127.21.28.66:37569)
I20260808 06:59:24.620200 7517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:54749
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:59:24.642349 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:53843/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46877
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:24.733644 8088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:24.733946 8088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:24.734050 8088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:24.735791 8088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:24.735900 8088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:24.735983 8088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:59:24.737787 8088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.28.84:53843/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:24.738014 8088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:24.738232 8088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:24.743052 8098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:24.743052 8100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:24.743397 8097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:24.743882 8088 server_base.cc:1034] running on GCE node
I20260808 06:59:24.744077 8088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:24.744287 8088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:24.747534 8088 hybrid_clock.cc:648] HybridClock initialized: now 1786172364747514 us; error 30 us; skew 500 ppm
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:24.750214 8088 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:59:24.750643 8088 webserver.cc:460] Webserver started at http://127.21.28.67:41683/ using document root <none> and password file <none>
I20260808 06:59:24.750875 8088 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:24.750923 8088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:24.751057 8088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:24.766602 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "41d45eb3bcc94f8781a1cedd7d47291c"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "b5ff88920ed7fa64c5f3b1096e3e3785"
server_key_iv: "dc757874d26d122ab2e1cd19eca1e87b"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.767046 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "41d45eb3bcc94f8781a1cedd7d47291c"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "b5ff88920ed7fa64c5f3b1096e3e3785"
server_key_iv: "dc757874d26d122ab2e1cd19eca1e87b"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.772089 8088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260808 06:59:24.788012 8114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.788229 8088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 06:59:24.788308 8088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "41d45eb3bcc94f8781a1cedd7d47291c"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "b5ff88920ed7fa64c5f3b1096e3e3785"
server_key_iv: "dc757874d26d122ab2e1cd19eca1e87b"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:24.788383 8088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:24.801374 8088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:24.803278 8088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:24.803431 8088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:24.803653 8088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:24.804095 8088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:24.804145 8088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.804178 8088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:24.804199 8088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:24.812933 8088 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:42587
I20260808 06:59:24.813295 8088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:59:24.814786 8303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:42587 every 8 connection(s)
I20260808 06:59:24.818153 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 8088
I20260808 06:59:24.818252 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:59:24.818439 21617 external_mini_cluster.cc:1428] Setting key 9fd5a2b824fdd04eefd99b2344141daf
Aug 08 06:59:24 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172364, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:24.823498 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:24.814558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:51993 (local address 127.21.28.126:46877)
0808 06:59:24.814608 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:24.814610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:24.814826 (+ 216us) server_negotiation.cc:407] Connection header received
0808 06:59:24.815523 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:24.815525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:24.815538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:24.815590 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:24.816213 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.816751 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.817715 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.817921 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.817975 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:24.820731 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:24.820733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:24.820733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:24.820740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:24.822061 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:24.822405 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:24.822406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:24.822406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:24.822469 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:24.822755 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:24.822757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:24.822757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:24.822898 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:24.822955 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:24.823308 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 06:59:24.823395 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:59:24.824308 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:24.814236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46877 (local address 127.21.28.67:51993)
0808 06:59:24.814692 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:24.814740 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:24.815416 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:24.815657 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:24.815661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:24.815917 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:24.816119 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:24.816125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.816835 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.816836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:24.817632 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:24.817635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:24.818589 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:24.818590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:24.818746 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:24.819371 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:24.819379 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:24.820648 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:24.822133 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:24.822139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:24.822141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:24.822340 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:24.822539 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:24.822540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:24.822540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:24.822674 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:24.823004 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:24.823010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:24.823162 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 06:59:24.824019 (+ 857us) client_negotiation.cc:240] Negotiation successful
0808 06:59:24.824145 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":85,"threads_started":1}
I20260808 06:59:24.824729 8304 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46877
I20260808 06:59:24.824817 8304 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:24.824995 8304 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:24.825333 7517 ts_manager.cc:194] Registered new tserver with Master: 41d45eb3bcc94f8781a1cedd7d47291c (127.21.28.67:42587)
I20260808 06:59:24.825701 7517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:51993
I20260808 06:59:24.829909 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:59:24.830601 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 7656
I20260808 06:59:24.864212 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 7865
I20260808 06:59:24.876020 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 8088
I20260808 06:59:24.892211 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 7466
I20260808 06:59:24.914039 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46877
--webserver_interface=127.21.28.126
--webserver_port=40467
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46877
--ranger_config_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:53843/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.28.84:53843/kms with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:24.994552 8308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:24.994740 8308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:24.994771 8308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:24.995918 8308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:24.995958 8308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:24.995977 8308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:24.995993 8308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:24.996019 8308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:24.997644 8308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.28.84:53843/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46877
--ranger_config_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:24.997879 8308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:24.998106 8308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:59:25.002687 8308 server_base.cc:1034] running on GCE node
W20260808 06:59:25.002599 8319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:25.002585 8318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:25.002602 8322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:25.003053 8308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:25.003310 8308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:25.004443 8308 hybrid_clock.cc:648] HybridClock initialized: now 1786172365004431 us; error 29 us; skew 500 ppm
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:25.006752 8308 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:59:25.007134 8308 webserver.cc:460] Webserver started at http://127.21.28.126:40467/ using document root <none> and password file <none>
I20260808 06:59:25.007308 8308 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:25.007354 8308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:25.049758 8308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:25.050467 8332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.050623 8308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:25.050709 8308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "6f488023438945048f59bf8df56a0aa2"
format_stamp: "Formatted at 2026-08-08 06:59:23 on dist-test-slave-nj21"
server_key: "67af5269c1d89b5f122df8cd1932f6bc"
server_key_iv: "6b0d1c8c01f8878a182d665e930b38c6"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:25.051064 8308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:25.068544 8308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:25.070367 8308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:25.070497 8308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:25.074611 8308 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46877
I20260808 06:59:25.074673 8417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46877 every 8 connection(s)
I20260808 06:59:25.075001 8308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:59:25.076262 8418 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:59:25.076924 8418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Bootstrap starting.
I20260808 06:59:25.079169 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 8308
I20260808 06:59:25.079435 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:53843/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:34433
--local_ip_for_outbound_sockets=127.21.28.65
--tserver_master_addrs=127.21.28.126:46877
--webserver_port=45095
--webserver_interface=127.21.28.65
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.28.84:53843/kms with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:59:25.085639 8418 log.cc:824] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:25.087281 8418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:59:25.087529 8418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Bootstrap complete.
I20260808 06:59:25.089804 8418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } }
I20260808 06:59:25.090030 8418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f488023438945048f59bf8df56a0aa2, State: Initialized, Role: FOLLOWER
I20260808 06:59:25.090178 8418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } }
I20260808 06:59:25.090251 8418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:59:25.090299 8418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:59:25.090345 8418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:59:25.091320 8418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } }
I20260808 06:59:25.091427 8418 leader_election.cc:302] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f488023438945048f59bf8df56a0aa2; no voters:
I20260808 06:59:25.091583 8418 leader_election.cc:288] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 06:59:25.091626 8422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:59:25.091763 8422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 6f488023438945048f59bf8df56a0aa2, State: Running, Role: LEADER
I20260808 06:59:25.091859 8422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } }
I20260808 06:59:25.091866 8418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:25.092124 8424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6f488023438945048f59bf8df56a0aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } } }
I20260808 06:59:25.092123 8425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f488023438945048f59bf8df56a0aa2. Latest consensus state: current_term: 2 leader_uuid: "6f488023438945048f59bf8df56a0aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f488023438945048f59bf8df56a0aa2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46877 } } }
I20260808 06:59:25.092228 8424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:25.092233 8425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:25.092437 8428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:59:25.092955 8428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:59:25.093086 8428 catalog_manager.cc:1259] Loaded cluster ID: b07db89639594294ae1cb6b12ac8abf1
I20260808 06:59:25.093123 8428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:59:25.093650 8418 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 06:59:25.097092 8428 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:59:25.097793 8428 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Loaded TSK: 0
I20260808 06:59:25.098078 8428 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:59:25.190889 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:25.191147 8420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:25.191186 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:25.192826 8420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:25.192890 8420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:25.192929 8420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:59:25.194983 8420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.28.84:53843/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:34433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=45095
--tserver_master_addrs=127.21.28.126:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:25.195216 8420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:25.195466 8420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:25.202135 8476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:25.202142 8477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:25.202946 8480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:25.203194 8420 server_base.cc:1034] running on GCE node
I20260808 06:59:25.203420 8420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:25.203708 8420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:25.204905 8420 hybrid_clock.cc:648] HybridClock initialized: now 1786172365204881 us; error 34 us; skew 500 ppm
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:25.208112 8420 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:59:25.208645 8420 webserver.cc:460] Webserver started at http://127.21.28.65:45095/ using document root <none> and password file <none>
I20260808 06:59:25.208927 8420 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:25.209010 8420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:25.240877 8420 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260808 06:59:25.241698 8492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.242302 8420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:25.242399 8420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "3ceab25a81d54e879dd9cb5a25374661"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "08b2560b4869510afb694232cd26f471"
server_key_iv: "3c567279a4d805ba2531a90cc091684a"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:25.242877 8420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:25.299947 8420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:25.301843 8420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:25.302000 8420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:25.302269 8420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:25.302636 8420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:25.302675 8420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.302707 8420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:25.302731 8420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.309288 8420 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:34433
I20260808 06:59:25.309593 8665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:34433 every 8 connection(s)
I20260808 06:59:25.309758 8420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:59:25.316411 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 8420
I20260808 06:59:25.316716 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:53843/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:37569
--local_ip_for_outbound_sockets=127.21.28.66
--tserver_master_addrs=127.21.28.126:46877
--webserver_port=38835
--webserver_interface=127.21.28.66
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.28.84:53843/kms with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:25.324067 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:25.310501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46877 (local address 127.21.28.65:40385)
0808 06:59:25.311021 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:25.311057 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:25.311625 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:25.312398 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:25.312403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:25.312631 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:25.312908 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:25.312916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.314680 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.314681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:25.315962 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:25.315965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.316817 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.316818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:25.316954 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:25.318472 (+ 1518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:25.318482 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:25.320121 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:25.322213 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:25.322220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:25.322222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:25.322501 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:25.322793 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:25.322794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:25.322794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:25.322922 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:25.323402 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:25.323409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:25.323563 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 06:59:25.323655 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 06:59:25.323838 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":110,"threads_started":1}
I20260808 06:59:25.324740 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:25.310600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:40385 (local address 127.21.28.126:46877)
0808 06:59:25.311911 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:25.311920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:25.311962 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:59:25.312113 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:25.312120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:25.312176 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:25.313049 (+ 873us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:25.313139 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.314556 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.316074 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.316688 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.318211 (+ 1523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:25.320250 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:25.320254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:25.320255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:25.320266 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:25.322101 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:25.322601 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:25.322603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:25.322604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:25.322708 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:25.322995 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:25.322996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:25.322996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:25.323307 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:25.324350 (+ 1043us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:25.324423 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 06:59:25.324540 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":72,"threads_started":1}
I20260808 06:59:25.325412 8667 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46877
I20260808 06:59:25.325506 8667 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.325681 8667 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
W20260808 06:59:25.326474 8667 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:46877 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 06:59:25.326539 8667 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.326606 8667 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:25.326985 8667 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.327061 8667 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
W20260808 06:59:25.327347 8667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:59:25.442505 8674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:25.442782 8674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:25.442819 8674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:25.444479 8674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:25.444538 8674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:25.444573 8674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:59:25.446627 8674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.28.84:53843/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:37569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=38835
--tserver_master_addrs=127.21.28.126:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:25.446923 8674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:25.447248 8674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:25.452486 8682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:25.452498 8685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:25.452669 8674 server_base.cc:1034] running on GCE node
W20260808 06:59:25.452494 8681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:25.453318 8674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:25.453569 8674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:25.454744 8674 hybrid_clock.cc:648] HybridClock initialized: now 1786172365454728 us; error 31 us; skew 500 ppm
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:25.457733 8674 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:59:25.458109 8674 webserver.cc:460] Webserver started at http://127.21.28.66:38835/ using document root <none> and password file <none>
I20260808 06:59:25.458334 8674 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:25.458401 8674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:25.478864 8674 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:25.480161 8699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.480368 8674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 06:59:25.480473 8674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "313098fe2c9c422b87aede7e5e87c8db"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "28624605baeb8c93a2efaa71f498af2c"
server_key_iv: "17ed3054ddedcfccb857509e647fa2c8"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:25.480929 8674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:25.493731 8674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:25.496035 8674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:25.496199 8674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:25.496462 8674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:25.496845 8674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:25.496908 8674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.496945 8674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:25.496968 8674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.509478 8674 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:37569
I20260808 06:59:25.509570 8875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:37569 every 8 connection(s)
I20260808 06:59:25.509974 8674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:59:25.510146 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 8674
I20260808 06:59:25.510444 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.28.84:53843/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:42587
--local_ip_for_outbound_sockets=127.21.28.67
--tserver_master_addrs=127.21.28.126:46877
--webserver_port=41683
--webserver_interface=127.21.28.67
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.28.84:53843/kms with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:25.523245 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:25.512106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:44147 (local address 127.21.28.126:46877)
0808 06:59:25.512157 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:25.512159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:25.513492 (+ 1333us) server_negotiation.cc:407] Connection header received
0808 06:59:25.514182 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:25.514184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:25.514197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:25.514227 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:25.515068 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.515870 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.517320 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.517618 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.517660 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:25.520020 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:25.520021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:25.520022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:25.520030 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:25.521588 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:25.522077 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:25.522079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:25.522079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:25.522139 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:25.522483 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:25.522484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:25.522485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:25.522622 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:25.522668 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:25.523064 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 06:59:25.523146 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 06:59:25.523965 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:25.512099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46877 (local address 127.21.28.66:44147)
0808 06:59:25.513351 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:25.513399 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:25.514047 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:25.514409 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:25.514413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:25.514644 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:25.514947 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:25.514955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.515987 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.515987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:25.517211 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:25.517214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.517787 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.517788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:25.517932 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:25.518453 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:25.518463 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:25.519905 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:25.521679 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:25.521686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:25.521688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:25.521984 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:25.522214 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:25.522215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:25.522216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:25.522400 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:25.522730 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:25.522737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:25.522899 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 06:59:25.523537 (+ 638us) client_negotiation.cc:240] Negotiation successful
0808 06:59:25.523732 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":84,"threads_started":1}
I20260808 06:59:25.524724 8876 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46877
I20260808 06:59:25.524829 8876 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.525018 8876 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
W20260808 06:59:25.525619 8876 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:46877 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 06:59:25.525687 8876 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.525758 8876 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:25.526064 8876 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.526127 8876 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
W20260808 06:59:25.526396 8876 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:59:25.613469 8878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:25.613724 8878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:25.613755 8878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:25.615413 8878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:25.615469 8878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:25.615505 8878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:59:25.617686 8878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.28.84:53843/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:42587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=41683
--tserver_master_addrs=127.21.28.126:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:25.617913 8878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:25.618145 8878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:25.624305 8896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:25.624500 8891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:25.624819 8890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:25.625361 8878 server_base.cc:1034] running on GCE node
I20260808 06:59:25.625592 8878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:25.625842 8878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:25.627020 8878 hybrid_clock.cc:648] HybridClock initialized: now 1786172365627001 us; error 32 us; skew 500 ppm
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:25.630384 8878 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:59:25.630967 8878 webserver.cc:460] Webserver started at http://127.21.28.67:41683/ using document root <none> and password file <none>
I20260808 06:59:25.631295 8878 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:25.631440 8878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:25.654026 8878 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:59:25.654795 8909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.654990 8878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:25.655076 8878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "41d45eb3bcc94f8781a1cedd7d47291c"
format_stamp: "Formatted at 2026-08-08 06:59:24 on dist-test-slave-nj21"
server_key: "b5ff88920ed7fa64c5f3b1096e3e3785"
server_key_iv: "dc757874d26d122ab2e1cd19eca1e87b"
server_key_version: "kuduclusterkey@0"
I20260808 06:59:25.656183 8878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:25.672570 8878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:25.674685 8878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:25.674829 8878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:25.675034 8878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:25.675390 8878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:25.675432 8878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.675460 8878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:25.675477 8878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:25.681998 8878 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:42587
I20260808 06:59:25.682433 9091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:42587 every 8 connection(s)
I20260808 06:59:25.682433 8878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:59:25.687117 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 8878
Aug 08 06:59:25 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:25.693410 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:25.683463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:33903 (local address 127.21.28.126:46877)
0808 06:59:25.683513 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:25.683515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:25.683944 (+ 429us) server_negotiation.cc:407] Connection header received
0808 06:59:25.684605 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:25.684606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:25.684617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:25.684658 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:25.685409 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.686087 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.687706 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.688019 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.688090 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:25.690258 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:25.690260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:25.690260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:25.690267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:25.691902 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:25.692371 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:25.692372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:25.692373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:25.692442 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:25.692730 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:25.692732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:25.692732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:25.692875 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:25.692951 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:25.693233 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 06:59:25.693317 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:59:25.693603 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:25.683471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46877 (local address 127.21.28.67:33903)
0808 06:59:25.683798 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:25.683846 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:25.684505 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:25.684738 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:25.684743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:25.684968 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:25.685238 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:25.685245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.686187 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.686188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:25.687454 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:25.687458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.688132 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.688133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:25.688276 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:25.688773 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:25.688781 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:25.690173 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:25.692007 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:25.692014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:25.692017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:25.692303 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:25.692512 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:25.692513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:25.692513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:25.692644 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:25.692954 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:25.692960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:25.693087 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 06:59:25.693183 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 06:59:25.693318 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":84,"threads_started":1}
I20260808 06:59:25.694113 9092 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46877
I20260808 06:59:25.694211 9092 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.694386 9092 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
W20260808 06:59:25.695051 9092 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:46877 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 06:59:25.695128 9092 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.695194 9092 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:25.695528 9092 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:25.695611 9092 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
W20260808 06:59:25.695955 9092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 06:59:25.696133 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:25.687516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.21.28.126:46877)
0808 06:59:25.687770 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:25.687771 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:25.687779 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:59:25.687800 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:25.687801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:25.687813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:25.687890 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:25.688504 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.689258 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.690614 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:25.690917 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:25.690985 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:25.692361 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:25.692363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:25.692363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:25.692370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:25.694872 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:25.695200 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:25.695201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:25.695201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:25.695261 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:25.695442 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:25.695443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:25.695443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:25.695548 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:25.695745 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:25.695890 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 06:59:25.695969 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":65,"threads_started":1}
Aug 08 06:59:26 dist-test-slave-nj21 krb5kdc[6756](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786172366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:26.327544 8667 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:26.327732 8667 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:26.328118 8365 ts_manager.cc:194] Registered new tserver with Master: 3ceab25a81d54e879dd9cb5a25374661 (127.21.28.65:34433)
I20260808 06:59:26.329149 8365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:40385
I20260808 06:59:26.526566 8876 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:26.526731 8876 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:26.527037 8365 ts_manager.cc:194] Registered new tserver with Master: 313098fe2c9c422b87aede7e5e87c8db (127.21.28.66:37569)
I20260808 06:59:26.527585 8365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:44147
Aug 08 06:59:26 dist-test-slave-nj21 krb5kdc[6756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for HTTP/127.21.28.84@KRBTEST.COM
I20260808 06:59:26.696131 9092 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:26.696282 9092 heartbeater.cc:511] Master 127.21.28.126:46877 requested a full tablet report, sending...
I20260808 06:59:26.696590 8365 ts_manager.cc:194] Registered new tserver with Master: 41d45eb3bcc94f8781a1cedd7d47291c (127.21.28.67:42587)
I20260808 06:59:26.697090 8365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:33903
I20260808 06:59:26.703627 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:59:26.711999 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.704744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.21.28.126:46877)
0808 06:59:26.706095 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:26.706097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:26.706105 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:59:26.706125 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:26.706125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:26.706136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:26.706208 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:26.706822 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.707322 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.708058 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.708266 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.708320 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.709533 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:26.709534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:26.709534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:26.709541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:26.710961 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:26.711269 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:26.711270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:26.711271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:26.711326 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:26.711488 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:26.711489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:26.711489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:26.711598 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:26.711713 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:26.711809 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 06:59:26.711893 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1332,"thread_start_us":71,"threads_started":1}
I20260808 06:59:26.713745 8365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:59:26.715708 8365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:59:26.723799 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.718468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:37569 (local address 127.0.0.1:35146)
0808 06:59:26.719291 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:26.719335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:26.719437 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:26.719718 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:26.719721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:26.719724 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:26.719800 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.719802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.720864 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.720865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.722220 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.722223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.723473 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.723474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.723554 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.723571 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 06:59:26.723638 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 06:59:26.723711 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":128,"threads_started":1}
I20260808 06:59:26.724136 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.719001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.21.28.66:37569)
0808 06:59:26.719134 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:26.719139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:26.719568 (+ 429us) server_negotiation.cc:407] Connection header received
0808 06:59:26.719606 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:26.719608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:26.719621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:26.719729 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:26.719903 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.720760 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.722326 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.723292 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.723359 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.723441 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:26.723980 (+ 539us) server_negotiation.cc:299] Negotiation successful
0808 06:59:26.724028 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260808 06:59:26.725150 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.718468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:42587 (local address 127.0.0.1:40352)
0808 06:59:26.720401 (+ 1933us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:26.720412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:26.720479 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:26.721009 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:26.721010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:26.721012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:26.721134 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.721137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.722444 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.722445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.723575 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.723579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.724713 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.724714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.724937 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.724944 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 06:59:26.725020 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 06:59:26.725065 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1900,"mutex_wait_us":4,"thread_start_us":128,"threads_started":1}
I20260808 06:59:26.725065 8769 tablet_service.cc:1467] Processing CreateTablet for tablet 880abf842e7c4f01b1d676f67e002bc2 (DEFAULT_TABLE table=test-table [id=52f3f6c536ec4c0e9baa30bf718cc700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:26.725512 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.718578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.21.28.67:42587)
0808 06:59:26.720296 (+ 1718us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:26.720302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:26.720712 (+ 410us) server_negotiation.cc:407] Connection header received
0808 06:59:26.720766 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:26.720769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:26.720786 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:26.720868 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:26.721223 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.722354 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.723695 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.724503 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.724618 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.724677 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:26.725346 (+ 669us) server_negotiation.cc:299] Negotiation successful
0808 06:59:26.725391 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1679,"thread_start_us":64,"threads_started":1}
I20260808 06:59:26.725546 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880abf842e7c4f01b1d676f67e002bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:26.728175 8962 tablet_service.cc:1467] Processing CreateTablet for tablet 880abf842e7c4f01b1d676f67e002bc2 (DEFAULT_TABLE table=test-table [id=52f3f6c536ec4c0e9baa30bf718cc700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:26.728538 8962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880abf842e7c4f01b1d676f67e002bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:26.730213 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.718750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:34433 (local address 127.0.0.1:42630)
0808 06:59:26.720599 (+ 1849us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:26.720609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:26.720652 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:26.726184 (+ 5532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:26.726185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:26.726187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:26.726232 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.726234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.727648 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.727649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.728771 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.728774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.729997 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.729999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.730062 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.730067 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:59:26.730103 (+ 36us) client_negotiation.cc:240] Negotiation successful
0808 06:59:26.730141 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"thread_start_us":102,"threads_started":1}
I20260808 06:59:26.730587 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.718900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.21.28.65:34433)
0808 06:59:26.725991 (+ 7091us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:26.725995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:26.726008 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:59:26.726045 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:26.726047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:26.726064 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:26.726319 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:26.726507 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.727530 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.728891 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.729879 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.730330 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.730398 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:26.730428 (+ 30us) server_negotiation.cc:299] Negotiation successful
0808 06:59:26.730471 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7052,"thread_start_us":62,"threads_started":1}
I20260808 06:59:26.732107 8564 tablet_service.cc:1467] Processing CreateTablet for tablet 880abf842e7c4f01b1d676f67e002bc2 (DEFAULT_TABLE table=test-table [id=52f3f6c536ec4c0e9baa30bf718cc700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:26.732487 8564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880abf842e7c4f01b1d676f67e002bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:26.734611 9151 tablet_bootstrap.cc:492] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Bootstrap starting.
I20260808 06:59:26.735922 9151 tablet_bootstrap.cc:654] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:26.736543 9151 log.cc:824] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:26.737442 9151 tablet_bootstrap.cc:492] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: No bootstrap required, opened a new log
I20260808 06:59:26.737525 9151 ts_tablet_manager.cc:1397] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:59:26.739845 9156 tablet_bootstrap.cc:492] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661: Bootstrap starting.
I20260808 06:59:26.740929 9156 tablet_bootstrap.cc:654] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:26.741075 9151 raft_consensus.cc:348] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.741271 9151 raft_consensus.cc:374] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:26.741331 9151 raft_consensus.cc:729] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313098fe2c9c422b87aede7e5e87c8db, State: Initialized, Role: FOLLOWER
I20260808 06:59:26.741487 9151 consensus_queue.cc:260] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.741542 9156 log.cc:824] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:26.741848 9151 ts_tablet_manager.cc:1428] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260808 06:59:26.741989 8876 heartbeater.cc:503] Master 127.21.28.126:46877 was elected leader, sending a full tablet report...
I20260808 06:59:26.741976 9154 tablet_bootstrap.cc:492] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: Bootstrap starting.
I20260808 06:59:26.743067 9154 tablet_bootstrap.cc:654] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:26.744236 9154 log.cc:824] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:26.744284 9156 tablet_bootstrap.cc:492] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661: No bootstrap required, opened a new log
I20260808 06:59:26.744357 9156 ts_tablet_manager.cc:1397] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 06:59:26.745275 9154 tablet_bootstrap.cc:492] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: No bootstrap required, opened a new log
I20260808 06:59:26.745359 9154 ts_tablet_manager.cc:1397] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 06:59:26.748944 9156 raft_consensus.cc:348] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.749127 9156 raft_consensus.cc:374] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:26.749178 9156 raft_consensus.cc:729] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ceab25a81d54e879dd9cb5a25374661, State: Initialized, Role: FOLLOWER
I20260808 06:59:26.749318 9156 consensus_queue.cc:260] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.749679 9156 ts_tablet_manager.cc:1428] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 06:59:26.749825 8667 heartbeater.cc:503] Master 127.21.28.126:46877 was elected leader, sending a full tablet report...
I20260808 06:59:26.750011 9154 raft_consensus.cc:348] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.750171 9154 raft_consensus.cc:374] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:26.750217 9154 raft_consensus.cc:729] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d45eb3bcc94f8781a1cedd7d47291c, State: Initialized, Role: FOLLOWER
I20260808 06:59:26.750350 9154 consensus_queue.cc:260] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.750699 9154 ts_tablet_manager.cc:1428] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260808 06:59:26.750813 9092 heartbeater.cc:503] Master 127.21.28.126:46877 was elected leader, sending a full tablet report...
I20260808 06:59:26.757149 9163 raft_consensus.cc:482] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:26.757292 9163 raft_consensus.cc:504] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.757782 9163 leader_election.cc:288] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 313098fe2c9c422b87aede7e5e87c8db (127.21.28.66:37569), 3ceab25a81d54e879dd9cb5a25374661 (127.21.28.65:34433)
W20260808 06:59:26.762313 8877 tablet.cc:2369] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:59:26.762802 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.758020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:55181 (local address 127.21.28.66:37569)
0808 06:59:26.758071 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:26.758073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:26.758494 (+ 421us) server_negotiation.cc:407] Connection header received
0808 06:59:26.758677 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:26.758678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:26.758701 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:26.758761 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:26.759164 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.759886 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.761660 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.762243 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.762302 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.762313 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:26.762675 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 06:59:26.762735 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 06:59:26.763180 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.758060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:37569 (local address 127.21.28.67:55181)
0808 06:59:26.758360 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:26.758403 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:26.758528 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:26.758861 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:26.758862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:26.758864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:26.759058 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.759061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.759996 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.759997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.761546 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.761549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.762465 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.762466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.762564 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.762570 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 06:59:26.762925 (+ 355us) client_negotiation.cc:240] Negotiation successful
0808 06:59:26.762970 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":38,"thread_start_us":142,"threads_started":1}
I20260808 06:59:26.763230 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.758518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:55835 (local address 127.21.28.65:34433)
0808 06:59:26.758553 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:26.758556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:26.758707 (+ 151us) server_negotiation.cc:407] Connection header received
0808 06:59:26.758832 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:26.758833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:26.758841 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:26.758866 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:26.759219 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.759997 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.761705 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.762248 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.762306 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.762315 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:26.763134 (+ 819us) server_negotiation.cc:299] Negotiation successful
0808 06:59:26.763175 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:59:26.763566 8796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880abf842e7c4f01b1d676f67e002bc2" candidate_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313098fe2c9c422b87aede7e5e87c8db" is_pre_election: true
I20260808 06:59:26.763844 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.758333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:34433 (local address 127.21.28.67:55835)
0808 06:59:26.758635 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:26.758645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:26.758742 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:26.758922 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:26.758923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:26.758925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:26.759137 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.759140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.760122 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.760122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.761609 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:26.761612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.762525 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.762527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:26.763047 (+ 520us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.763052 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:59:26.763604 (+ 552us) client_negotiation.cc:240] Negotiation successful
0808 06:59:26.763644 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"thread_start_us":112,"threads_started":1}
I20260808 06:59:26.763824 8796 raft_consensus.cc:2454] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41d45eb3bcc94f8781a1cedd7d47291c in term 0.
I20260808 06:59:26.764139 8913 leader_election.cc:302] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313098fe2c9c422b87aede7e5e87c8db, 41d45eb3bcc94f8781a1cedd7d47291c; no voters:
I20260808 06:59:26.764344 9163 raft_consensus.cc:2781] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:59:26.764410 9163 raft_consensus.cc:482] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:59:26.764443 9163 raft_consensus.cc:3037] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:26.764648 8594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880abf842e7c4f01b1d676f67e002bc2" candidate_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ceab25a81d54e879dd9cb5a25374661" is_pre_election: true
I20260808 06:59:26.764825 8594 raft_consensus.cc:2454] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41d45eb3bcc94f8781a1cedd7d47291c in term 0.
I20260808 06:59:26.765695 9163 raft_consensus.cc:504] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.765833 9163 leader_election.cc:288] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [CANDIDATE]: Term 1 election: Requested vote from peers 313098fe2c9c422b87aede7e5e87c8db (127.21.28.66:37569), 3ceab25a81d54e879dd9cb5a25374661 (127.21.28.65:34433)
I20260808 06:59:26.766047 8594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880abf842e7c4f01b1d676f67e002bc2" candidate_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ceab25a81d54e879dd9cb5a25374661"
I20260808 06:59:26.766076 8796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880abf842e7c4f01b1d676f67e002bc2" candidate_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313098fe2c9c422b87aede7e5e87c8db"
I20260808 06:59:26.766121 8594 raft_consensus.cc:3037] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:26.766150 8796 raft_consensus.cc:3037] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:26.767338 8796 raft_consensus.cc:2454] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d45eb3bcc94f8781a1cedd7d47291c in term 1.
I20260808 06:59:26.767481 8594 raft_consensus.cc:2454] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d45eb3bcc94f8781a1cedd7d47291c in term 1.
I20260808 06:59:26.767504 8913 leader_election.cc:302] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313098fe2c9c422b87aede7e5e87c8db, 41d45eb3bcc94f8781a1cedd7d47291c; no voters:
I20260808 06:59:26.767724 9163 raft_consensus.cc:2781] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:26.767872 9163 raft_consensus.cc:686] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 1 LEADER]: Becoming Leader. State: Replica: 41d45eb3bcc94f8781a1cedd7d47291c, State: Running, Role: LEADER
I20260808 06:59:26.767959 9163 consensus_queue.cc:237] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } }
I20260808 06:59:26.768679 8365 catalog_manager.cc:5495] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c reported cstate change: term changed from 0 to 1, leader changed from <none> to 41d45eb3bcc94f8781a1cedd7d47291c (127.21.28.67). New cstate: current_term: 1 leader_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41d45eb3bcc94f8781a1cedd7d47291c" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 42587 } health_report { overall_health: HEALTHY } } }
I20260808 06:59:26.777755 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.774454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.21.28.67:42587)
0808 06:59:26.774490 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:26.774492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:26.774534 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:59:26.774604 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:26.774605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:26.774621 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:26.774679 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:59:26.774881 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.775567 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.776716 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:26.777015 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:26.777076 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:26.777284 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:59:26.777405 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:59:26.777480 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:26.777664 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 06:59:26.777684 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:59:26.780191 8594 raft_consensus.cc:1261] T 880abf842e7c4f01b1d676f67e002bc2 P 3ceab25a81d54e879dd9cb5a25374661 [term 1 FOLLOWER]: Refusing update from remote peer 41d45eb3bcc94f8781a1cedd7d47291c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:26.780323 8796 raft_consensus.cc:1261] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 1 FOLLOWER]: Refusing update from remote peer 41d45eb3bcc94f8781a1cedd7d47291c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:26.780555 9163 consensus_queue.cc:1035] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ceab25a81d54e879dd9cb5a25374661" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 34433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:26.780818 9166 consensus_queue.cc:1035] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [LEADER]: Connected to new peer: Peer: permanent_uuid: "313098fe2c9c422b87aede7e5e87c8db" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:26.783869 9175 mvcc.cc:205] Tried to move back new op lower bound from 7316162014329282560 to 7316162014282133504. Current Snapshot: MvccSnapshot[applied={T|T < 7316162014329282560}]
I20260808 06:59:26.784214 9173 mvcc.cc:205] Tried to move back new op lower bound from 7316162014329282560 to 7316162014282133504. Current Snapshot: MvccSnapshot[applied={T|T < 7316162014329282560}]
I20260808 06:59:26.786021 9177 mvcc.cc:205] Tried to move back new op lower bound from 7316162014329282560 to 7316162014282133504. Current Snapshot: MvccSnapshot[applied={T|T < 7316162014329282560}]
I20260808 06:59:26.788934 8365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:59:26.789021 8365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:59:26.789672 8365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f488023438945048f59bf8df56a0aa2: Sending DeleteTablet for 3 replicas of tablet 880abf842e7c4f01b1d676f67e002bc2
I20260808 06:59:26.790035 8962 tablet_service.cc:1514] Processing DeleteTablet for tablet 880abf842e7c4f01b1d676f67e002bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:26 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:40352
I20260808 06:59:26.790431 8564 tablet_service.cc:1514] Processing DeleteTablet for tablet 880abf842e7c4f01b1d676f67e002bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:26 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:42630
I20260808 06:59:26.790637 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 8420
I20260808 06:59:26.790867 8769 tablet_service.cc:1514] Processing DeleteTablet for tablet 880abf842e7c4f01b1d676f67e002bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:26 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:35146
I20260808 06:59:26.798197 9188 tablet_replica.cc:330] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: stopping tablet replica
I20260808 06:59:26.798372 9188 raft_consensus.cc:2229] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:59:26.798533 9188 raft_consensus.cc:2258] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:26.799031 9188 ts_tablet_manager.cc:1905] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:26.801057 9188 ts_tablet_manager.cc:1918] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:26.801177 9188 log.cc:1196] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/880abf842e7c4f01b1d676f67e002bc2
I20260808 06:59:26.801486 9188 ts_tablet_manager.cc:1939] T 880abf842e7c4f01b1d676f67e002bc2 P 313098fe2c9c422b87aede7e5e87c8db: Deleting consensus metadata
I20260808 06:59:26.801946 8339 catalog_manager.cc:4841] TS 313098fe2c9c422b87aede7e5e87c8db (127.21.28.66:37569): tablet 880abf842e7c4f01b1d676f67e002bc2 (table test-table [id=52f3f6c536ec4c0e9baa30bf718cc700]) successfully deleted
I20260808 06:59:26.811830 9186 tablet_replica.cc:330] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: stopping tablet replica
I20260808 06:59:26.811977 9186 raft_consensus.cc:2229] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 1 LEADER]: Raft consensus shutting down.
W20260808 06:59:26.812238 8339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:34433 (error 108)
I20260808 06:59:26.812260 9186 raft_consensus.cc:2258] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:26.812812 9186 ts_tablet_manager.cc:1905] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:26.812862 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 8674
I20260808 06:59:26.815733 9186 ts_tablet_manager.cc:1918] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:26.815820 9186 log.cc:1196] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/880abf842e7c4f01b1d676f67e002bc2
I20260808 06:59:26.816100 9186 ts_tablet_manager.cc:1939] T 880abf842e7c4f01b1d676f67e002bc2 P 41d45eb3bcc94f8781a1cedd7d47291c: Deleting consensus metadata
I20260808 06:59:26.816509 8336 catalog_manager.cc:4841] TS 41d45eb3bcc94f8781a1cedd7d47291c (127.21.28.67:42587): tablet 880abf842e7c4f01b1d676f67e002bc2 (table test-table [id=52f3f6c536ec4c0e9baa30bf718cc700]) successfully deleted
I20260808 06:59:26.912276 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:26.912052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:34433 (local address 127.0.0.1:42638)
0808 06:59:26.912125 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:26.912209 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:34433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260808 06:59:26.912482 8339 catalog_manager.cc:4584] TS 3ceab25a81d54e879dd9cb5a25374661 (127.21.28.65:34433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 880abf842e7c4f01b1d676f67e002bc2: Network error: Client connection negotiation failed: client connection to 127.21.28.65:34433: connect: Connection refused (error 111)
I20260808 06:59:26.914227 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 8878
I20260808 06:59:26.923918 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 8308
I20260808 06:59:26.934257 21617 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 06:59:27.277567 21617 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 06:59:27.277650 21617 mini_ranger.cc:66] Stopping Ranger...
I20260808 06:59:27.620707 21617 mini_ranger.cc:68] Stopped Ranger
2026-08-08 06:59:27.620 UTC [6805] LOG: received smart shutdown request
2026-08-08 06:59:27.625 UTC [6805] LOG: background worker "logical replication launcher" (PID 6821) exited with exit code 1
2026-08-08 06:59:27.625 UTC [6815] LOG: shutting down
2026-08-08 06:59:27.753 UTC [6805] LOG: database system is shut down
2026-08-08T06:59:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47185 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:59:27 dist-test-slave-nj21 krb5kdc[9202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:59:27 dist-test-slave-nj21 krb5kdc[9202](info): set up 2 sockets
Aug 08 06:59:27 dist-test-slave-nj21 krb5kdc[9202](info): commencing operation
krb5kdc: starting...
W20260808 06:59:28.969867 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:59:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:59:29.077821 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36829
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36829,127.21.28.125:44099,127.21.28.124:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:29.155599 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:29.155799 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:29.155819 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:29.156862 9225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:29.156894 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:29.156906 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:29.156917 9225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:29.156931 9225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:29.158324 9225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36829,127.21.28.125:44099,127.21.28.124:34037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:29.158502 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:29.158666 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:29.163069 9235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:29.163079 9236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:29.163214 9225 server_base.cc:1034] running on GCE node
W20260808 06:59:29.163113 9240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:29.163504 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:29.163800 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:29.164943 9225 hybrid_clock.cc:648] HybridClock initialized: now 1786172369164924 us; error 32 us; skew 500 ppm
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:29.167272 9225 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:59:29.167662 9225 webserver.cc:460] Webserver started at http://127.21.28.126:33667/ using document root <none> and password file <none>
I20260808 06:59:29.167878 9225 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:29.167922 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:29.168008 9225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:29.168970 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "5f732f4a47d947618d975487c9435db2"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "09cc666e4619179371d53890b0c1b721"
server_key_iv: "5a435227d791d6ce54c7575369b4e1de"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.169255 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "5f732f4a47d947618d975487c9435db2"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "09cc666e4619179371d53890b0c1b721"
server_key_iv: "5a435227d791d6ce54c7575369b4e1de"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.170447 9225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 06:59:29.171129 9250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:29.171306 9225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:59:29.171404 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "5f732f4a47d947618d975487c9435db2"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "09cc666e4619179371d53890b0c1b721"
server_key_iv: "5a435227d791d6ce54c7575369b4e1de"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.171461 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:29.183651 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:29.185714 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:29.185845 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:29.189555 9225 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:36829
I20260808 06:59:29.189599 9331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:36829 every 8 connection(s)
I20260808 06:59:29.189896 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:59:29.190667 9332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:29.191605 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 9225
I20260808 06:59:29.191733 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:59:29.191937 21617 external_mini_cluster.cc:1428] Setting key 23e64c446c333db95bff12ba9aeb9d0b
I20260808 06:59:29.192487 9332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.28.125@KRBTEST.COM; defaulting to no policy
I20260808 06:59:29.201838 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.193173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36829 (local address 127.0.0.1:59268)
0808 06:59:29.193347 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.193387 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.193884 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.194158 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.194161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.194386 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.194539 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.194543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.195937 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.195938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.196618 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.196620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.197261 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.197262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.197312 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.197794 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.197802 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.199101 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.200716 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.200722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.200724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.200901 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.201102 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.201102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.201102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.201169 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.201441 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.201443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.201490 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.201553 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.201666 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260808 06:59:29.201841 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.193206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.21.28.126:36829)
0808 06:59:29.193373 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.193382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.193484 (+ 102us) server_negotiation.cc:407] Connection header received
0808 06:59:29.194035 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.194037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.194059 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.194148 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.194625 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.195852 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.196700 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.197187 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.197301 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.199181 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.199190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.199190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.199198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.200643 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.200974 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.200976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.200976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.201053 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.201210 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.201210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.201211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.201397 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.201452 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.201605 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.201737 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.125@KRBTEST.COM" created.
I20260808 06:59:29.202553 9332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:59:29.202857 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.202741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45392)
0808 06:59:29.202784 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.202833 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 06:59:29.202993 9253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
I20260808 06:59:29.203357 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.203273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45408)
0808 06:59:29.203312 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.203338 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 06:59:29.203449 9332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:44099: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab.
Entry for principal kudu/127.21.28.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab.
I20260808 06:59:29.214249 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.125:44099
--webserver_interface=127.21.28.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab
--principal=kudu/127.21.28.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36829,127.21.28.125:44099,127.21.28.124:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:59:29.252640 9332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } attempt: 1
I20260808 06:59:29.253053 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.252942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45412)
0808 06:59:29.252997 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.253025 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 06:59:29.253328 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.253255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45428)
0808 06:59:29.253288 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.253311 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260808 06:59:29.253425 9332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:44099: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
W20260808 06:59:29.292729 9342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:29.292923 9342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:29.292954 9342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:29.294040 9342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:29.294080 9342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:29.294097 9342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:29.294113 9342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:29.294128 9342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:29.295608 9342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36829,127.21.28.125:44099,127.21.28.124:34037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab
--principal=kudu/127.21.28.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.125:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.28.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:29.295854 9342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:29.296060 9342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:29.300474 9353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:29.300469 9354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:29.300484 9362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:29.300732 9342 server_base.cc:1034] running on GCE node
I20260808 06:59:29.300948 9342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:29.301198 9342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:29.302327 9342 hybrid_clock.cc:648] HybridClock initialized: now 1786172369302318 us; error 26 us; skew 500 ppm
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:29.304697 9342 init.cc:377] Logged in from keytab as kudu/127.21.28.125@KRBTEST.COM (short username kudu)
I20260808 06:59:29.305074 9342 webserver.cc:460] Webserver started at http://127.21.28.125:36683/ using document root <none> and password file <none>
I20260808 06:59:29.305274 9342 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:29.305320 9342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:29.305446 9342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:29.306370 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/data/instance:
uuid: "68ce1075b90c4ae3a7d56eddec899c3d"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "6a9abec226e98b18d0330a7d277f14ce"
server_key_iv: "c76512b5ed4a46484b4c9fd6cc83b5dc"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.306686 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/wal/instance:
uuid: "68ce1075b90c4ae3a7d56eddec899c3d"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "6a9abec226e98b18d0330a7d277f14ce"
server_key_iv: "c76512b5ed4a46484b4c9fd6cc83b5dc"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.307880 9342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:59:29.308498 9369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:29.308640 9342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:29.308720 9342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/wal
uuid: "68ce1075b90c4ae3a7d56eddec899c3d"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "6a9abec226e98b18d0330a7d277f14ce"
server_key_iv: "c76512b5ed4a46484b4c9fd6cc83b5dc"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.308784 9342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:29.321553 9332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } attempt: 2
I20260808 06:59:29.321909 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.321785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45440)
0808 06:59:29.321837 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.321872 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 06:59:29.322178 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.322112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45452)
0808 06:59:29.322141 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.322163 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260808 06:59:29.322258 9332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:44099: Network error: Client connection negotiation failed: client connection to 127.21.28.125:44099: connect: Connection refused (error 111)
I20260808 06:59:29.337775 9342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:29.342171 9342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:29.342283 9342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:29.346058 9342 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.125:44099
I20260808 06:59:29.346088 9448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.125:44099 every 8 connection(s)
I20260808 06:59:29.346397 9342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
I20260808 06:59:29.347188 9449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:29.348244 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 9342
I20260808 06:59:29.348336 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-1/wal/instance
I20260808 06:59:29.348506 21617 external_mini_cluster.cc:1428] Setting key 40b094e80cc3a132fa1920570d553ee4
I20260808 06:59:29.348805 9449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.28.124@KRBTEST.COM; defaulting to no policy
I20260808 06:59:29.357082 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.349485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.21.28.126:36829)
0808 06:59:29.349533 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.349535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.349755 (+ 220us) server_negotiation.cc:407] Connection header received
0808 06:59:29.350210 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.350212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.350229 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.350277 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.350863 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.351344 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.352342 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.352548 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.352606 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.354547 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.354548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.354549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.354555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.355846 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.356198 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.356199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.356200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.356257 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.356472 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.356473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.356473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.356600 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.356660 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.356909 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.356994 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 06:59:29.357213 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.349484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36829 (local address 127.0.0.1:59282)
0808 06:59:29.349640 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.349674 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.350115 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.350348 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.350351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.350589 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.350776 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.350780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.351428 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.351430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.352262 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.352264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.352618 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.352618 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.352731 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.353179 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.353186 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.354448 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.355916 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.355924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.355926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.356132 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.356309 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.356310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.356310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.356417 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.356649 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.356656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.356787 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.356874 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.357014 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260808 06:59:29.357774 9449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.124@KRBTEST.COM" created.
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 06:59:29.365103 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.358014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45462)
0808 06:59:29.358058 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.358069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.358110 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.358315 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.358316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.358438 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.358487 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.358489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.359617 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.359618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.360339 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.360342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.360863 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.360864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.360936 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.361239 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.361244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.362493 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.364112 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.364113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.364113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.364304 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.364488 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.364489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.364489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.364555 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.364777 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.364778 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.364823 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.364897 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.364987 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 06:59:29.365127 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.358067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.21.28.125:44099)
0808 06:59:29.358186 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.358190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.358209 (+ 19us) server_negotiation.cc:407] Connection header received
0808 06:59:29.358228 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.358230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.358239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.358301 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.358553 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.359523 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.360424 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.360786 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.360842 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.362570 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.362577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.362578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.362583 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.364038 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.364388 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.364390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.364390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.364436 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.364610 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.364610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.364610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.364721 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.364775 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.364919 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.365033 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260808 06:59:29.365697 9449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:59:29.365971 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.365882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45588)
0808 06:59:29.365913 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.365953 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260808 06:59:29.366093 9375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
I20260808 06:59:29.366425 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.366355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45592)
0808 06:59:29.366387 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.366409 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260808 06:59:29.366544 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34037: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab.
Entry for principal kudu/127.21.28.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab.
I20260808 06:59:29.371342 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.124:34037
--webserver_interface=127.21.28.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab
--principal=kudu/127.21.28.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36829,127.21.28.125:44099,127.21.28.124:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:59:29.413398 9332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } attempt: 3
I20260808 06:59:29.415732 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } attempt: 1
I20260808 06:59:29.416744 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.416624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45598)
0808 06:59:29.416679 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.416705 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 06:59:29.417100 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.417020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45614)
0808 06:59:29.417052 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.417076 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260808 06:59:29.417325 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34037: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 06:59:29.423224 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.413736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45464)
0808 06:59:29.413793 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.413804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.413851 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.413994 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.413995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.414147 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.414211 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.414213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.414908 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.414909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.415464 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.415468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.416689 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.416690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.416799 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.417313 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.417318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.420664 (+ 3346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.422338 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.422339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.422339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.422551 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.422732 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.422732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.422732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.422777 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.422989 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.422990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.423029 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.423066 (+ 37us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.423147 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 06:59:29.423465 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.413803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.21.28.125:44099)
0808 06:59:29.413862 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.413864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.413875 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:59:29.413923 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.413924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.413936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.413983 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.414283 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.414815 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.415556 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.416329 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.416414 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.420737 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.420739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.420739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.420747 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.422257 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.422623 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.422624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.422624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.422677 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.422824 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.422824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.422825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.422936 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.423308 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.423339 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.423401 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:59:29.425323 9332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:59:29.425648 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.425552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45626)
0808 06:59:29.425595 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.425619 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 06:59:29.425981 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.425888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45632)
0808 06:59:29.425928 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.425956 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 06:59:29.426061 9332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34037: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
I20260808 06:59:29.457180 9332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } attempt: 1
I20260808 06:59:29.457612 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.457501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45638)
0808 06:59:29.457555 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.457589 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 06:59:29.457917 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.457827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45652)
0808 06:59:29.457868 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.457893 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 06:59:29.457998 9332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34037: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
W20260808 06:59:29.463572 9459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:29.463871 9459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:29.463954 9459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:29.465610 9459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:29.465696 9459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:29.465749 9459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:29.465797 9459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:29.465844 9459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:29.468088 9459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36829,127.21.28.125:44099,127.21.28.124:34037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab
--principal=kudu/127.21.28.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.124:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.28.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:29.468457 9459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:29.468715 9459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:29.473886 9469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:29.473930 9467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:29.473932 9464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:29.474172 9459 server_base.cc:1034] running on GCE node
I20260808 06:59:29.474391 9459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:29.474699 9459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:29.475833 9459 hybrid_clock.cc:648] HybridClock initialized: now 1786172369475820 us; error 30 us; skew 500 ppm
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:29.478080 9459 init.cc:377] Logged in from keytab as kudu/127.21.28.124@KRBTEST.COM (short username kudu)
I20260808 06:59:29.478504 9459 webserver.cc:460] Webserver started at http://127.21.28.124:35627/ using document root <none> and password file <none>
I20260808 06:59:29.478704 9459 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:29.478750 9459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:29.478873 9459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:29.479909 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/data/instance:
uuid: "51f263ff827541fea688cfb6fec8bdb5"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "55959be4dc234f6e08ec3fef8e989bba"
server_key_iv: "f600e5e8d3edb21eb386ec0c09de0281"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.480222 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/wal/instance:
uuid: "51f263ff827541fea688cfb6fec8bdb5"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "55959be4dc234f6e08ec3fef8e989bba"
server_key_iv: "f600e5e8d3edb21eb386ec0c09de0281"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.481377 9459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:29.481999 9485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:29.482142 9459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:29.482235 9459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/wal
uuid: "51f263ff827541fea688cfb6fec8bdb5"
format_stamp: "Formatted at 2026-08-08 06:59:29 on dist-test-slave-nj21"
server_key: "55959be4dc234f6e08ec3fef8e989bba"
server_key_iv: "f600e5e8d3edb21eb386ec0c09de0281"
server_key_version: "encryptionkey@0"
I20260808 06:59:29.482293 9459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:29.485526 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } attempt: 2
I20260808 06:59:29.485888 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.485771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45662)
0808 06:59:29.485819 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.485847 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 06:59:29.486156 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.486079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45674)
0808 06:59:29.486112 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.486134 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260808 06:59:29.486234 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34037: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34037: connect: Connection refused (error 111)
I20260808 06:59:29.514709 9459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:29.516381 9459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:29.516517 9459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:29.520246 9459 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.124:34037
I20260808 06:59:29.520282 9575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.124:34037 every 8 connection(s)
I20260808 06:59:29.520588 9459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
I20260808 06:59:29.521373 9576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:29.522842 9576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:59:29.525949 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 9459
I20260808 06:59:29.526031 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786172218768454-21617-0/minicluster-data/master-2/wal/instance
I20260808 06:59:29.526187 21617 external_mini_cluster.cc:1428] Setting key 7fbfb1cef609654422c615c5a4b2b190
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:29.531255 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.523496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.21.28.126:36829)
0808 06:59:29.523554 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.523556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.523754 (+ 198us) server_negotiation.cc:407] Connection header received
0808 06:59:29.524197 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.524198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.524207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.524258 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.524783 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.525242 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.526165 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.526386 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.526448 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.528638 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.528640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.528640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.528647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.529997 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.530420 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.530421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.530422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.530479 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.530687 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.530687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.530688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.530823 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.530890 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.531107 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.531189 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:59:29.531391 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.523457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36829 (local address 127.0.0.1:59292)
0808 06:59:29.523617 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.523654 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.524127 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.524309 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.524312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.524540 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.524720 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.524723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.525296 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.525297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.526077 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.526080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.526464 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.526465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.526609 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.527032 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.527042 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.528572 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.530074 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.530079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.530081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.530349 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.530544 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.530544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.530545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.530636 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.530889 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.530892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.530992 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.531065 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.531181 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
I20260808 06:59:29.531980 9576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:59:29.533128 9332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } attempt: 2
I20260808 06:59:29.535398 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.526514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.21.28.124:34037)
0808 06:59:29.526694 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.526699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.526723 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:59:29.526772 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.526774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.526794 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.526857 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.527421 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.528435 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.529386 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.529721 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.529791 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.532127 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.532134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.532134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.532141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.533814 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.534158 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.534161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.534161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.534216 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.534415 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.534416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.534416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.534580 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.534735 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.535242 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.535309 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
I20260808 06:59:29.538519 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.532215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:44099 (local address 127.0.0.1:45480)
0808 06:59:29.532329 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.532340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.532381 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.532493 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.532494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.532654 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.532710 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.532712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.533467 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.533468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.533941 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.533943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.534330 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.534331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.534399 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.534755 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.534759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.535996 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.537526 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.537526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.537527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.537722 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.537922 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.537923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.537924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.537991 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.538224 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.538225 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.538264 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.538335 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.538429 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260808 06:59:29.538553 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.532283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.21.28.125:44099)
0808 06:59:29.532328 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.532330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.532371 (+ 41us) server_negotiation.cc:407] Connection header received
0808 06:59:29.532427 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.532429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.532439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.532491 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.532780 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.533381 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.534010 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.534280 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.534335 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.536077 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.536078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.536079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.536086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.537446 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.537799 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.537800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.537800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.537855 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.538051 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.538051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.538052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.538167 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.538221 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.538380 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.538464 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 06:59:29.538846 9576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:29.541397 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.533337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45692)
0808 06:59:29.533378 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.533386 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.533420 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.534142 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.534143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.534277 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.534314 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.534316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.535050 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.535051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.535808 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.535811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.536191 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.536192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.536251 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.536677 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.536682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.537987 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.540436 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.540437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.540438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.540634 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.540822 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.540822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.540823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.540894 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.541118 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.541119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.541161 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.541224 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.541308 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 06:59:29.541705 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.533397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.21.28.124:34037)
0808 06:59:29.534041 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.534042 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.534046 (+ 4us) server_negotiation.cc:407] Connection header received
0808 06:59:29.534059 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.534059 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.534066 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.534115 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.534410 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.534978 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.535878 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.536127 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.536189 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.538250 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.538251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.538251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.538256 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.540350 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.540698 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.540698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.540699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.540758 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.540953 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.540954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.540954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.541061 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.541116 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.541515 (+ 399us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.541584 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":75,"threads_started":1}
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
I20260808 06:59:29.543464 9332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2: Bootstrap starting.
I20260808 06:59:29.544236 9332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:29.544436 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.536520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.21.28.126:36829)
0808 06:59:29.536557 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.536559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.536723 (+ 164us) server_negotiation.cc:407] Connection header received
0808 06:59:29.536777 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.536778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.536787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.536818 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.537477 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.538077 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.539070 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.539294 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.539347 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.541626 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.541627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.541628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.541635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.543344 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.543635 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.543636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.543636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.543714 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.543927 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.543928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.543929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.544033 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.544161 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.544289 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.544361 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:59:29.544556 9332 log.cc:824] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:29.545436 9332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2: No bootstrap required, opened a new log
I20260808 06:59:29.546092 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.539052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45700)
0808 06:59:29.539395 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.539405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.539448 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.539587 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.539588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.539744 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.539776 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.539778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.540553 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.540554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.541178 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.541180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.541475 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.541476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.541516 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.542021 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.542026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.543142 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.545044 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.545045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.545046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.545278 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.545484 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.545485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.545485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.545541 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.545798 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.545799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.545842 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.545912 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.546000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260808 06:59:29.546146 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.539123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.21.28.124:34037)
0808 06:59:29.539175 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.539177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.539457 (+ 280us) server_negotiation.cc:407] Connection header received
0808 06:59:29.539497 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.539498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.539508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.539574 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.539845 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.540497 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.541244 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.541419 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.541641 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.543223 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.543224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.543224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.543229 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.544949 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.545357 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.545358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.545358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.545421 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.545600 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.545601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.545601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.545726 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.545789 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.545929 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.546048 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":19}
I20260808 06:59:29.547122 9576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: Bootstrap starting.
I20260808 06:59:29.547411 9332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.547519 9332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:29.547551 9332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f732f4a47d947618d975487c9435db2, State: Initialized, Role: FOLLOWER
I20260808 06:59:29.547708 9332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.547792 9576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:29.547952 9591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.548027 9591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:59:29.548066 9576 log.cc:824] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:29.548223 9332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:29.548851 9576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: No bootstrap required, opened a new log
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
W20260808 06:59:29.550417 9608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:59:29.550518 9608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:59:29.550743 9576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.550873 9576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:29.550930 9576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51f263ff827541fea688cfb6fec8bdb5, State: Initialized, Role: FOLLOWER
I20260808 06:59:29.551043 9576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.551285 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.551324 9576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:29.551414 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 06:59:29.553320 9641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:59:29.553416 9641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:59:29.551386 9225 master_runner.cc:186] Error getting master registration for 127.21.28.125:44099: OK, instance_id { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" instance_seqno: 1786172369345030 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 06:59:29.554286 9225 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 06:59:29.554068 9459 master_runner.cc:186] Error getting master registration for 127.21.28.125:44099: OK, instance_id { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" instance_seqno: 1786172369345030 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 06:59:29.556321 9459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 06:59:29.556756 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.545331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.21.28.125:44099)
0808 06:59:29.545363 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.545364 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.545373 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:59:29.545391 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.545392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.545400 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.545444 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.545854 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.546486 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.547345 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.547602 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.547667 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.550061 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.550062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.550062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.550069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.555041 (+ 4972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.555417 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.555419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.555419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.555475 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.555655 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.555656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.555657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.555793 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.556374 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.556585 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.556664 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:59:29.557974 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:59:29.567799 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.559808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.21.28.125:44099)
0808 06:59:29.559842 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.559844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.560216 (+ 372us) server_negotiation.cc:407] Connection header received
0808 06:59:29.560307 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.560308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.560317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.560343 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.561158 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.561825 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.562827 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.563103 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.563170 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.564428 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.564430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.564430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.564437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.566493 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.566995 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.566997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.566997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.567052 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.567248 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.567249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.567250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.567364 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.567428 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.567620 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.567715 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 06:59:29.567848 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.559455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.21.28.126:36829)
0808 06:59:29.559496 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.559498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.559607 (+ 109us) server_negotiation.cc:407] Connection header received
0808 06:59:29.559748 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.559749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.559759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.559786 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.560508 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.561191 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.562194 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.562484 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.562522 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.564146 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.564147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.564148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.564154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.565809 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.567003 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.567004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.567004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.567058 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.567276 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.567277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.567277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.567392 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.567478 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.567690 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.567761 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 06:59:29.577350 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } attempt: 3
I20260808 06:59:29.578028 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.560036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.21.28.124:34037)
0808 06:59:29.560076 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.560078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.567796 (+ 7718us) server_negotiation.cc:407] Connection header received
0808 06:59:29.567885 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.567886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.567897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.567926 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.571875 (+ 3949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.572587 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.573517 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.573833 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.573896 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.575269 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.575270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.575270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.575277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.576804 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.577172 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.577173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.577173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.577226 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.577453 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.577454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.577455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.577582 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.577645 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.577849 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.577943 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 08 06:59:29 dist-test-slave-nj21 krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172369, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
I20260808 06:59:29.586210 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.577595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34037 (local address 127.0.0.1:45722)
0808 06:59:29.577668 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:29.577677 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:29.577707 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:29.577929 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:29.577929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:29.578077 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:29.578116 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.578117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.579764 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.579766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.580541 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:29.580543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.581561 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.581562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:29.581622 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.582000 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:29.582004 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:29.583388 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:29.585199 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.585200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.585201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.585397 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.585637 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:29.585638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:29.585639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:29.585694 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:29.585937 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:29.585938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:29.585977 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 06:59:29.586033 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 06:59:29.586118 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 06:59:29.586571 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:29.577765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.21.28.124:34037)
0808 06:59:29.577798 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:29.577800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:29.577806 (+ 6us) server_negotiation.cc:407] Connection header received
0808 06:59:29.577822 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:29.577822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:29.577850 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:29.577894 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:29.578361 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.579073 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.580623 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:29.580933 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:29.581014 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:29.583485 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:29.583486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:29.583487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:29.583493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:29.585067 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.585503 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.585504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.585504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.585555 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:29.585751 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:29.585752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:29.585752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:29.585862 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:29.585913 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:29.586419 (+ 506us) server_negotiation.cc:299] Negotiation successful
0808 06:59:29.586491 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:59:29.588251 9449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d: Bootstrap starting.
I20260808 06:59:29.588953 9449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:29.589228 9449 log.cc:824] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:29.590083 9449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d: No bootstrap required, opened a new log
I20260808 06:59:29.592181 9449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.592361 9449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:29.592453 9449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ce1075b90c4ae3a7d56eddec899c3d, State: Initialized, Role: FOLLOWER
I20260808 06:59:29.592631 9449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.592993 9449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:29.592955 9665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.593176 9665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [sys.catalog]: This master's current role is: FOLLOWER
W20260808 06:59:29.594822 9679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:59:29.594934 9679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:59:29.596573 9342 master_runner.cc:263] No leader master found from master 68ce1075b90c4ae3a7d56eddec899c3d
I20260808 06:59:29.596618 9342 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 06:59:29.686535 9262 tablet.cc:2369] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:59:29.767073 9494 tablet.cc:2369] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:59:29.820500 9591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:29.820606 9591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.820801 9591 leader_election.cc:288] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ce1075b90c4ae3a7d56eddec899c3d (127.21.28.125:44099), 51f263ff827541fea688cfb6fec8bdb5 (127.21.28.124:34037)
I20260808 06:59:29.821053 9411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f732f4a47d947618d975487c9435db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" is_pre_election: true
I20260808 06:59:29.821058 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f732f4a47d947618d975487c9435db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51f263ff827541fea688cfb6fec8bdb5" is_pre_election: true
I20260808 06:59:29.821179 9411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f732f4a47d947618d975487c9435db2 in term 0.
I20260808 06:59:29.821193 9535 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f732f4a47d947618d975487c9435db2 in term 0.
I20260808 06:59:29.821342 9253 leader_election.cc:302] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f732f4a47d947618d975487c9435db2, 68ce1075b90c4ae3a7d56eddec899c3d; no voters:
I20260808 06:59:29.821496 9591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:59:29.821559 9591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:59:29.821574 9591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:29.822281 9591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.822397 9591 leader_election.cc:288] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [CANDIDATE]: Term 1 election: Requested vote from peers 68ce1075b90c4ae3a7d56eddec899c3d (127.21.28.125:44099), 51f263ff827541fea688cfb6fec8bdb5 (127.21.28.124:34037)
I20260808 06:59:29.822525 9411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f732f4a47d947618d975487c9435db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ce1075b90c4ae3a7d56eddec899c3d"
I20260808 06:59:29.822540 9535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f732f4a47d947618d975487c9435db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51f263ff827541fea688cfb6fec8bdb5"
I20260808 06:59:29.822598 9411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:29.822618 9535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:29.823243 9411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f732f4a47d947618d975487c9435db2 in term 1.
I20260808 06:59:29.823375 9535 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f732f4a47d947618d975487c9435db2 in term 1.
I20260808 06:59:29.823424 9253 leader_election.cc:302] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f732f4a47d947618d975487c9435db2, 68ce1075b90c4ae3a7d56eddec899c3d; no voters:
I20260808 06:59:29.823508 9591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:29.823612 9591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [term 1 LEADER]: Becoming Leader. State: Replica: 5f732f4a47d947618d975487c9435db2, State: Running, Role: LEADER
I20260808 06:59:29.823714 9591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } }
I20260808 06:59:29.824108 9682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f732f4a47d947618d975487c9435db2. Latest consensus state: current_term: 1 leader_uuid: "5f732f4a47d947618d975487c9435db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.824182 9682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:29.824378 9686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:59:29.824558 9686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:59:29.825776 9411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [term 1 FOLLOWER]: Refusing update from remote peer 5f732f4a47d947618d975487c9435db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:29.825853 9535 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [term 1 FOLLOWER]: Refusing update from remote peer 5f732f4a47d947618d975487c9435db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:29.826056 9682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:29.826157 9591 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:29.827227 9665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f732f4a47d947618d975487c9435db2. Latest consensus state: current_term: 1 leader_uuid: "5f732f4a47d947618d975487c9435db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.827322 9665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:59:29.827404 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f732f4a47d947618d975487c9435db2. Latest consensus state: current_term: 1 leader_uuid: "5f732f4a47d947618d975487c9435db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.827461 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:59:29.828029 9591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f732f4a47d947618d975487c9435db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.828104 9591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:29.828145 9665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f732f4a47d947618d975487c9435db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.828222 9665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:59:29.828562 9686 catalog_manager.cc:1347] Generated new cluster ID: 5b3290af9d2d4024b787a55a1f0f3114
I20260808 06:59:29.828611 9686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:59:29.828887 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f732f4a47d947618d975487c9435db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.828956 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:59:29.828477 9702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f732f4a47d947618d975487c9435db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f732f4a47d947618d975487c9435db2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36829 } } peers { permanent_uuid: "68ce1075b90c4ae3a7d56eddec899c3d" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 44099 } } peers { permanent_uuid: "51f263ff827541fea688cfb6fec8bdb5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34037 } } }
I20260808 06:59:29.831815 9702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:29.839511 9686 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:59:29.840078 9686 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:59:29.844558 9686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f732f4a47d947618d975487c9435db2: Generated new TSK 0
I20260808 06:59:29.844759 9686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:59:30.553849 9641 catalog_manager.cc:1259] Loaded cluster ID: 5b3290af9d2d4024b787a55a1f0f3114
I20260808 06:59:30.553926 9641 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: loading cluster ID for follower catalog manager: success
I20260808 06:59:30.554719 9641 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: acquiring CA information for follower catalog manager: success
I20260808 06:59:30.555009 9641 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51f263ff827541fea688cfb6fec8bdb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:59:30.595376 9679 catalog_manager.cc:1259] Loaded cluster ID: 5b3290af9d2d4024b787a55a1f0f3114
I20260808 06:59:30.595456 9679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d: loading cluster ID for follower catalog manager: success
I20260808 06:59:30.596244 9679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d: acquiring CA information for follower catalog manager: success
I20260808 06:59:30.596508 9679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68ce1075b90c4ae3a7d56eddec899c3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:59:30.903555 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 9225
I20260808 06:59:30.918335 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 9342
I20260808 06:59:30.925056 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 9459
2026-08-08T06:59:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3080 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:59:30 dist-test-slave-nj21 krb5kdc[9716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:59:30 dist-test-slave-nj21 krb5kdc[9716](info): set up 2 sockets
Aug 08 06:59:30 dist-test-slave-nj21 krb5kdc[9716](info): commencing operation
krb5kdc: starting...
W20260808 06:59:32.066649 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:59:32 dist-test-slave-nj21 krb5kdc[9716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:59:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:59:32.178318 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:35921
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:32.256516 9734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:32.256704 9734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:32.256734 9734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:32.257827 9734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:32.257866 9734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:32.257885 9734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:32.257901 9734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:32.257916 9734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:32.259377 9734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:35921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:32.259603 9734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:32.259819 9734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:32.264542 9742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:32.264556 9744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:32.264551 9746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:32.264748 9734 server_base.cc:1034] running on GCE node
I20260808 06:59:32.264978 9734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:32.265241 9734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:32.266382 9734 hybrid_clock.cc:648] HybridClock initialized: now 1786172372266367 us; error 28 us; skew 500 ppm
Aug 08 06:59:32 dist-test-slave-nj21 krb5kdc[9716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172372, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:32.268723 9734 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:59:32.269107 9734 webserver.cc:460] Webserver started at http://127.21.28.126:41157/ using document root <none> and password file <none>
I20260808 06:59:32.269306 9734 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:32.269352 9734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:32.269484 9734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:32.270432 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "bba23502693a45d1a77ea3087509e3fa"
format_stamp: "Formatted at 2026-08-08 06:59:32 on dist-test-slave-nj21"
server_key: "c807260f200f1148279cc1d39099dfac"
server_key_iv: "45837a0d1dc2e1398d86155b67f46691"
server_key_version: "encryptionkey@0"
I20260808 06:59:32.270737 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "bba23502693a45d1a77ea3087509e3fa"
format_stamp: "Formatted at 2026-08-08 06:59:32 on dist-test-slave-nj21"
server_key: "c807260f200f1148279cc1d39099dfac"
server_key_iv: "45837a0d1dc2e1398d86155b67f46691"
server_key_version: "encryptionkey@0"
I20260808 06:59:32.271975 9734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 06:59:32.272573 9755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:32.272722 9734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:59:32.272804 9734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "bba23502693a45d1a77ea3087509e3fa"
format_stamp: "Formatted at 2026-08-08 06:59:32 on dist-test-slave-nj21"
server_key: "c807260f200f1148279cc1d39099dfac"
server_key_iv: "45837a0d1dc2e1398d86155b67f46691"
server_key_version: "encryptionkey@0"
I20260808 06:59:32.272859 9734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:32.286501 9734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:32.288093 9734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:32.288215 9734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:32.291934 9734 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:35921
I20260808 06:59:32.291981 9835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:35921 every 8 connection(s)
I20260808 06:59:32.292270 9734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:59:32.293061 9836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:32.295145 9836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa: Bootstrap starting.
I20260808 06:59:32.295643 9836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:32.295930 9836 log.cc:824] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:32.296515 9836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa: No bootstrap required, opened a new log
I20260808 06:59:32.297994 9836 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba23502693a45d1a77ea3087509e3fa" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35921 } }
I20260808 06:59:32.298118 9836 raft_consensus.cc:374] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:32.298144 9836 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba23502693a45d1a77ea3087509e3fa, State: Initialized, Role: FOLLOWER
I20260808 06:59:32.298251 9836 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba23502693a45d1a77ea3087509e3fa" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35921 } }
I20260808 06:59:32.298300 9836 raft_consensus.cc:388] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:59:32.298321 9836 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:59:32.298350 9836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:32.298786 9836 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba23502693a45d1a77ea3087509e3fa" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35921 } }
I20260808 06:59:32.298863 9836 leader_election.cc:302] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bba23502693a45d1a77ea3087509e3fa; no voters:
I20260808 06:59:32.298974 9836 leader_election.cc:288] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:59:32.299017 9841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:32.299175 9836 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:32.299176 9841 raft_consensus.cc:686] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [term 1 LEADER]: Becoming Leader. State: Replica: bba23502693a45d1a77ea3087509e3fa, State: Running, Role: LEADER
I20260808 06:59:32.299356 9841 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba23502693a45d1a77ea3087509e3fa" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35921 } }
I20260808 06:59:32.299868 9843 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bba23502693a45d1a77ea3087509e3fa. Latest consensus state: current_term: 1 leader_uuid: "bba23502693a45d1a77ea3087509e3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba23502693a45d1a77ea3087509e3fa" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35921 } } }
I20260808 06:59:32.299953 9843 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:32.300175 9842 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bba23502693a45d1a77ea3087509e3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba23502693a45d1a77ea3087509e3fa" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35921 } } }
I20260808 06:59:32.300384 9842 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:32.300431 9857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:59:32.300863 9857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:59:32.302102 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 9734
I20260808 06:59:32.302177 9857 catalog_manager.cc:1347] Generated new cluster ID: 0c9997e4f935417880b036cb32abf38a
I20260808 06:59:32.302209 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:59:32.302219 9857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:59:32.302431 21617 external_mini_cluster.cc:1428] Setting key e22d0c250a253b620db6ebf9bab3f586
Aug 08 06:59:32 dist-test-slave-nj21 krb5kdc[9716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:32.314373 9857 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:59:32.314826 9857 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:59:32.316193 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:32.302826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.21.28.126:35921)
0808 06:59:32.302972 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:32.302980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:32.303995 (+ 1015us) server_negotiation.cc:407] Connection header received
0808 06:59:32.304110 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:32.304116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:32.304178 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:32.304313 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:32.305035 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:32.306739 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:32.307824 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:32.308436 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:32.308521 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:32.311183 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:32.311188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:32.311188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:32.311199 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:32.313030 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:32.313407 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:32.313409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:32.313410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:32.313523 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:32.313702 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:32.313703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:32.313703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:32.313960 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:32.314098 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:32.315797 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0808 06:59:32.315947 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260808 06:59:32.317159 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:59:32.323168 9857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bba23502693a45d1a77ea3087509e3fa: Generated new TSK 0
I20260808 06:59:32.323321 9857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:59:32.370846 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 9734
2026-08-08T06:59:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1445 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:59:32 dist-test-slave-nj21 krb5kdc[9878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:59:32 dist-test-slave-nj21 krb5kdc[9878](info): set up 2 sockets
Aug 08 06:59:32 dist-test-slave-nj21 krb5kdc[9878](info): commencing operation
krb5kdc: starting...
W20260808 06:59:33.493803 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:59:33 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:59:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:59:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:59:33.600155 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36283
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:33.680825 9901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:33.681090 9901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:33.681125 9901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:33.682855 9901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:33.682916 9901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:33.682934 9901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:33.682951 9901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:33.682966 9901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:33.684661 9901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:33.684890 9901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:33.685101 9901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:33.689680 9907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:33.689742 9908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:33.689776 9901 server_base.cc:1034] running on GCE node
W20260808 06:59:33.689705 9911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:33.690063 9901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:33.690344 9901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:33.691485 9901 hybrid_clock.cc:648] HybridClock initialized: now 1786172373691473 us; error 29 us; skew 500 ppm
Aug 08 06:59:33 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172373, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:33.693945 9901 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:59:33.694350 9901 webserver.cc:460] Webserver started at http://127.21.28.126:45913/ using document root <none> and password file <none>
I20260808 06:59:33.694535 9901 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:33.694571 9901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:33.694657 9901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:33.695603 9901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "14cd4072046a40ebbdad734ed393884b"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "a60aab80b08924ec351c4ee7d54f80bd"
server_key_iv: "202ccd74e685d6600863feb436b97271"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.695941 9901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "14cd4072046a40ebbdad734ed393884b"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "a60aab80b08924ec351c4ee7d54f80bd"
server_key_iv: "202ccd74e685d6600863feb436b97271"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.697077 9901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:33.697670 9923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:33.697805 9901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:33.697886 9901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "14cd4072046a40ebbdad734ed393884b"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "a60aab80b08924ec351c4ee7d54f80bd"
server_key_iv: "202ccd74e685d6600863feb436b97271"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.697942 9901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:33.705075 9901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:33.706931 9901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:33.707052 9901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:33.710767 9901 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:36283
I20260808 06:59:33.710801 10003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:36283 every 8 connection(s)
I20260808 06:59:33.711081 9901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:59:33.711972 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:33.713716 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 9901
I20260808 06:59:33.713819 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:59:33.713995 21617 external_mini_cluster.cc:1428] Setting key 8c2081aa9aa30ec61f3664cdff65aa97
I20260808 06:59:33.714097 10005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Bootstrap starting.
I20260808 06:59:33.714789 10005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:33.715103 10005 log.cc:824] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:33.715818 10005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: No bootstrap required, opened a new log
I20260808 06:59:33.717983 10005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:33.718116 10005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:33.718150 10005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cd4072046a40ebbdad734ed393884b, State: Initialized, Role: FOLLOWER
I20260808 06:59:33.718264 10005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:33.718348 10005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:59:33.718384 10005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:59:33.718436 10005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:33.719053 10005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:33.719148 10005 leader_election.cc:302] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14cd4072046a40ebbdad734ed393884b; no voters:
I20260808 06:59:33.719298 10005 leader_election.cc:288] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:59:33.719329 10012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:33.719435 10012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 1 LEADER]: Becoming Leader. State: Replica: 14cd4072046a40ebbdad734ed393884b, State: Running, Role: LEADER
I20260808 06:59:33.719525 10012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:33.719553 10005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:33.719887 10012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cd4072046a40ebbdad734ed393884b. Latest consensus state: current_term: 1 leader_uuid: "14cd4072046a40ebbdad734ed393884b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } } }
I20260808 06:59:33.719981 10012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:33.719898 10013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14cd4072046a40ebbdad734ed393884b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } } }
I20260808 06:59:33.720119 10013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: This master's current role is: LEADER
Aug 08 06:59:33 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:33.721071 10021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:59:33.721573 10021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:59:33.723132 10021 catalog_manager.cc:1347] Generated new cluster ID: 68b7936c95c14d73847ef045fb65920b
I20260808 06:59:33.723181 10021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:59:33.724792 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:33.714419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.21.28.126:36283)
0808 06:59:33.714580 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:33.714589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:33.714625 (+ 36us) server_negotiation.cc:407] Connection header received
0808 06:59:33.715187 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:33.715196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:33.715254 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:33.715384 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:33.716083 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:33.717402 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:33.718113 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:33.718685 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:33.718791 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:33.721257 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:33.721261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:33.721261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:33.721272 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:33.723014 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:33.723366 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:33.723369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:33.723369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:33.723460 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:33.723623 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:33.723624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:33.723624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:33.723896 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:33.724040 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:33.724476 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 06:59:33.724595 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260808 06:59:33.732045 10021 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:59:33.732533 10021 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:59:33.736111 10021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Generated new TSK 0
I20260808 06:59:33.736251 10021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:59:33.760375 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36283
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:33.839046 10054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:33.839236 10054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:33.839267 10054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:33.840415 10054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:33.840454 10054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:33.840488 10054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:59:33.841876 10054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:33.842088 10054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:33.842283 10054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:33.846901 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:33.846961 10061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:33.846909 10067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:33.847096 10054 server_base.cc:1034] running on GCE node
I20260808 06:59:33.847271 10054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:33.847481 10054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:33.848609 10054 hybrid_clock.cc:648] HybridClock initialized: now 1786172373848595 us; error 27 us; skew 500 ppm
Aug 08 06:59:33 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172373, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:33.850906 10054 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:59:33.851323 10054 webserver.cc:460] Webserver started at http://127.21.28.65:46489/ using document root <none> and password file <none>
I20260808 06:59:33.851516 10054 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:33.851562 10054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:33.851706 10054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:33.852656 10054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "138de8155d4142f3acf3c215f4de8b12"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "4973b456fc69cda36e9d45feff07b42d"
server_key_iv: "be64b624ad8f2f11def2287cb958fba2"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.852969 10054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "138de8155d4142f3acf3c215f4de8b12"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "4973b456fc69cda36e9d45feff07b42d"
server_key_iv: "be64b624ad8f2f11def2287cb958fba2"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.854135 10054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 06:59:33.854758 10076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:33.854900 10054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:59:33.854985 10054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "138de8155d4142f3acf3c215f4de8b12"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "4973b456fc69cda36e9d45feff07b42d"
server_key_iv: "be64b624ad8f2f11def2287cb958fba2"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.855042 10054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:33.864828 10054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:33.866338 10054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:33.866480 10054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:33.866672 10054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:33.867018 10054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:33.867080 10054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:33.867112 10054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:33.867131 10054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:33.872898 10054 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:33111
I20260808 06:59:33.872936 10257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:33111 every 8 connection(s)
I20260808 06:59:33.873195 10054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:59:33.874861 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 10054
I20260808 06:59:33.874963 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:59:33.875159 21617 external_mini_cluster.cc:1428] Setting key 63599e7cd643e78944b76fd4d52d9e07
Aug 08 06:59:33 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172373, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:33.882472 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:33.873822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:46145 (local address 127.21.28.126:36283)
0808 06:59:33.873882 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:33.873884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:33.874092 (+ 208us) server_negotiation.cc:407] Connection header received
0808 06:59:33.874508 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:33.874509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:33.874520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:33.874575 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:33.875544 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:33.876181 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:33.877450 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:33.877756 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:33.877830 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:33.879797 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:33.879799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:33.879799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:33.879806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:33.881247 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:33.881603 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:33.881604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:33.881604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:33.881668 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:33.881870 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:33.881871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:33.881872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:33.882010 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:33.882079 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:33.882278 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 06:59:33.882375 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 06:59:33.882515 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:33.873789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:46145)
0808 06:59:33.873945 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:33.873985 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:33.874437 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:33.874622 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:33.874624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:33.875231 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:33.875455 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:33.875461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:33.876268 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:33.876269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:33.877362 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:33.877366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:33.877834 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:33.877835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:33.877933 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:33.878391 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:33.878398 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:33.879711 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:33.881323 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:33.881329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:33.881331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:33.881535 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:33.881726 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:33.881727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:33.881727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:33.881818 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:33.882066 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:33.882069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:33.882168 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 06:59:33.882249 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 06:59:33.882362 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260808 06:59:33.882989 10258 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36283
I20260808 06:59:33.883073 10258 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:33.883251 10258 heartbeater.cc:511] Master 127.21.28.126:36283 requested a full tablet report, sending...
I20260808 06:59:33.883790 9947 ts_manager.cc:194] Registered new tserver with Master: 138de8155d4142f3acf3c215f4de8b12 (127.21.28.65:33111)
I20260808 06:59:33.884368 9947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:46145
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:59:33.899367 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36283
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:33.976939 10265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:33.977124 10265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:33.977154 10265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:33.978240 10265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:33.978278 10265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:33.978312 10265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:59:33.979660 10265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:33.979912 10265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:33.980103 10265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:33.984607 10275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:33.984614 10279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:33.984616 10274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:33.984767 10265 server_base.cc:1034] running on GCE node
I20260808 06:59:33.985052 10265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:33.985247 10265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:33.986387 10265 hybrid_clock.cc:648] HybridClock initialized: now 1786172373986367 us; error 27 us; skew 500 ppm
Aug 08 06:59:33 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172373, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:33.988727 10265 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:59:33.989096 10265 webserver.cc:460] Webserver started at http://127.21.28.66:35841/ using document root <none> and password file <none>
I20260808 06:59:33.989254 10265 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:33.989290 10265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:33.989374 10265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:33.990278 10265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "083b9226e90b4e72a1621b1a4603e586"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "fe5432b386f0c00cd3ff217c0e19d907"
server_key_iv: "5703cca9e971c2045a29997ee0b41a79"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.990566 10265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "083b9226e90b4e72a1621b1a4603e586"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "fe5432b386f0c00cd3ff217c0e19d907"
server_key_iv: "5703cca9e971c2045a29997ee0b41a79"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.991664 10265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:59:33.992306 10295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:33.992456 10265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:33.992537 10265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "083b9226e90b4e72a1621b1a4603e586"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "fe5432b386f0c00cd3ff217c0e19d907"
server_key_iv: "5703cca9e971c2045a29997ee0b41a79"
server_key_version: "encryptionkey@0"
I20260808 06:59:33.992596 10265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:34.001612 10265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:34.003489 10265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:34.003613 10265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:34.003826 10265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:34.004086 10265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:34.004117 10265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.004143 10265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:34.004166 10265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.009819 10265 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40403
I20260808 06:59:34.009871 10466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40403 every 8 connection(s)
I20260808 06:59:34.010120 10265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:59:34.013459 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 10265
I20260808 06:59:34.013554 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:59:34.013752 21617 external_mini_cluster.cc:1428] Setting key d47e1899acdaea26f9d50b562433f32d
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172373, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:34.019560 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.010705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:48697 (local address 127.21.28.126:36283)
0808 06:59:34.010746 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.010748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.010929 (+ 181us) server_negotiation.cc:407] Connection header received
0808 06:59:34.011373 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.011374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.011386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.011436 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:34.012528 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.013212 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.014650 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.014937 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.014992 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.017130 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:34.017131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:34.017132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:34.017138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:34.018344 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.018693 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.018694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.018694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.018751 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.018956 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.018957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.018958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.019067 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:34.019123 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.019366 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.019450 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:59:34.019709 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.010671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:48697)
0808 06:59:34.010829 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.010861 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.011285 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.011497 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.011501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.012149 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:34.012397 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.012403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.013300 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.013301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.014556 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.014559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.015008 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.015010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.015133 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.015548 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:34.015555 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:34.017040 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:34.018422 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.018429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.018431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.018629 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.018804 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.018805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.018806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.018901 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.019118 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:34.019124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:34.019249 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.019333 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.019490 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260808 06:59:34.020287 10467 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36283
I20260808 06:59:34.020370 10467 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:34.020526 10467 heartbeater.cc:511] Master 127.21.28.126:36283 requested a full tablet report, sending...
I20260808 06:59:34.020874 9947 ts_manager.cc:194] Registered new tserver with Master: 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403)
I20260808 06:59:34.021220 9947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:48697
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:59:34.035964 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36283
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:34.114235 10473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:34.114430 10473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:34.114461 10473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:34.115566 10473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:34.115605 10473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:34.115639 10473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:59:34.117050 10473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:34.117259 10473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:34.117465 10473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:34.122205 10484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.122244 10480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.122290 10473 server_base.cc:1034] running on GCE node
W20260808 06:59:34.122229 10481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.122526 10473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:34.122735 10473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:34.123873 10473 hybrid_clock.cc:648] HybridClock initialized: now 1786172374123862 us; error 30 us; skew 500 ppm
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:34.126082 10473 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:59:34.126482 10473 webserver.cc:460] Webserver started at http://127.21.28.67:44879/ using document root <none> and password file <none>
I20260808 06:59:34.126667 10473 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:34.126713 10473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:34.126830 10473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:34.127817 10473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "526e76d4ea62488384543c3b398e060d"
format_stamp: "Formatted at 2026-08-08 06:59:34 on dist-test-slave-nj21"
server_key: "34ff11ef621b8f727a8df2b1cebf0dbf"
server_key_iv: "349e9647a61de739055e9e5b25d8e424"
server_key_version: "encryptionkey@0"
I20260808 06:59:34.128134 10473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "526e76d4ea62488384543c3b398e060d"
format_stamp: "Formatted at 2026-08-08 06:59:34 on dist-test-slave-nj21"
server_key: "34ff11ef621b8f727a8df2b1cebf0dbf"
server_key_iv: "349e9647a61de739055e9e5b25d8e424"
server_key_version: "encryptionkey@0"
I20260808 06:59:34.129295 10473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:59:34.129933 10497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.130081 10473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:59:34.130162 10473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "526e76d4ea62488384543c3b398e060d"
format_stamp: "Formatted at 2026-08-08 06:59:34 on dist-test-slave-nj21"
server_key: "34ff11ef621b8f727a8df2b1cebf0dbf"
server_key_iv: "349e9647a61de739055e9e5b25d8e424"
server_key_version: "encryptionkey@0"
I20260808 06:59:34.130218 10473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:34.143574 10473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:34.145200 10473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:34.145327 10473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:34.145521 10473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:34.145790 10473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:34.145823 10473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.145859 10473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:34.145884 10473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.151549 10473 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:35863
I20260808 06:59:34.151588 10677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:35863 every 8 connection(s)
I20260808 06:59:34.151889 10473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:34.159837 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.152521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:42481 (local address 127.21.28.126:36283)
0808 06:59:34.152569 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.152571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.152748 (+ 177us) server_negotiation.cc:407] Connection header received
0808 06:59:34.153204 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.153205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.153214 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.153262 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:34.154094 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.154589 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.155481 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.155656 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.155769 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.157718 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:34.157719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:34.157720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:34.157725 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:34.158826 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.159159 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.159159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.159159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.159202 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.159368 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.159369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.159369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.159459 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:34.159501 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.159713 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.159774 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:59:34.160073 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 10473
I20260808 06:59:34.160157 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:59:34.160323 21617 external_mini_cluster.cc:1428] Setting key 1ed53bc54831a55850a7d89be4952795
I20260808 06:59:34.160566 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.152490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:42481)
0808 06:59:34.152644 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.152676 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.153131 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.153311 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.153314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.153846 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:34.154029 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.154033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.154651 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.154651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.155423 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.155425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.155782 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.155782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.155890 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.156321 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:34.156329 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:34.157651 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:34.158892 (+ 1241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.158898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.158899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.159104 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.159243 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.159244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.159244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.159329 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.159499 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:34.159502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:34.159603 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.159699 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.159821 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260808 06:59:34.160950 10679 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36283
I20260808 06:59:34.161041 10679 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:34.161217 10679 heartbeater.cc:511] Master 127.21.28.126:36283 requested a full tablet report, sending...
I20260808 06:59:34.161543 9947 ts_manager.cc:194] Registered new tserver with Master: 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
I20260808 06:59:34.161906 9947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:42481
I20260808 06:59:34.171975 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:59:34.178517 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.173308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.21.28.126:36283)
0808 06:59:34.173372 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.173374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.173461 (+ 87us) server_negotiation.cc:407] Connection header received
0808 06:59:34.173523 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.173523 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.173533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.173577 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:34.174128 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.174605 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.175289 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.175489 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.175541 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.176561 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:34.176563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:34.176563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:34.176570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:34.177520 (+ 950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.177803 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.177804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.177804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.177856 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.178016 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.178017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.178017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.178157 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:34.178244 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.178353 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.178431 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 06:59:34.179972 9947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:59:34.181571 9947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:59:34.188017 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.183508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34796 (local address 127.21.28.67:35863)
0808 06:59:34.183668 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.183699 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.184002 (+ 303us) server_negotiation.cc:407] Connection header received
0808 06:59:34.184078 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.184081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.184096 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.184160 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.184383 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.185473 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.186955 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.187629 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.187719 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.187781 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.187904 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.187942 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260808 06:59:34.188020 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.183458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35863 (local address 127.0.0.1:34796)
0808 06:59:34.183891 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.183931 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.184000 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.184262 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.184263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.184265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.184317 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.184319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.185565 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.185565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.186873 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.186875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.187710 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.187710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.187793 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.187807 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.187874 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.187934 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":29,"thread_start_us":57,"threads_started":1}
I20260808 06:59:34.188829 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.183459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40403 (local address 127.0.0.1:36596)
0808 06:59:34.184098 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.184105 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.184151 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.184533 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.184534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.184535 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.184564 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.184565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.185688 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.185689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.186873 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.186875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.188589 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.188590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.188651 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.188655 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.188698 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.188740 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":15,"thread_start_us":95,"threads_started":1}
I20260808 06:59:34.188845 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.183721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.21.28.66:40403)
0808 06:59:34.183897 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.183901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.184400 (+ 499us) server_negotiation.cc:407] Connection header received
0808 06:59:34.184447 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.184450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.184465 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.184534 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.184597 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.185598 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.187736 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.188446 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.188504 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.188558 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.188724 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.188764 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260808 06:59:34.188823 10577 tablet_service.cc:1467] Processing CreateTablet for tablet f8a4cb36bc8f4178adf1e9c428aa793e (DEFAULT_TABLE table=test-table [id=c3e005b049d8453a823844c13bfa09fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:34.189200 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.183463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:33111 (local address 127.0.0.1:40660)
0808 06:59:34.183943 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.183953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.184012 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.184251 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.184254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.184257 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.184313 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.184315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.186514 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.186515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.187396 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.187399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.188627 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.188628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.188689 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.188694 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.188739 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.188779 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":109,"threads_started":1}
I20260808 06:59:34.189217 10577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a4cb36bc8f4178adf1e9c428aa793e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:34.189215 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.183752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.21.28.65:33111)
0808 06:59:34.183919 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.183924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.183999 (+ 75us) server_negotiation.cc:407] Connection header received
0808 06:59:34.184073 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.184076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.184092 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.184164 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.185599 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.186435 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.187480 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.188474 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.188539 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.188594 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.188873 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.189140 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260808 06:59:34.189541 10367 tablet_service.cc:1467] Processing CreateTablet for tablet f8a4cb36bc8f4178adf1e9c428aa793e (DEFAULT_TABLE table=test-table [id=c3e005b049d8453a823844c13bfa09fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:34.189873 10367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a4cb36bc8f4178adf1e9c428aa793e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:34.189859 10151 tablet_service.cc:1467] Processing CreateTablet for tablet f8a4cb36bc8f4178adf1e9c428aa793e (DEFAULT_TABLE table=test-table [id=c3e005b049d8453a823844c13bfa09fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:34.190161 10151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a4cb36bc8f4178adf1e9c428aa793e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:34.191710 10703 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: Bootstrap starting.
I20260808 06:59:34.192306 10703 tablet_bootstrap.cc:654] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:34.192703 10703 log.cc:824] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:34.193109 10706 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: Bootstrap starting.
I20260808 06:59:34.193384 10703 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: No bootstrap required, opened a new log
I20260808 06:59:34.193449 10703 ts_tablet_manager.cc:1397] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:59:34.193830 10706 tablet_bootstrap.cc:654] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:34.194353 10706 log.cc:824] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:34.194705 10707 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Bootstrap starting.
I20260808 06:59:34.195086 10706 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: No bootstrap required, opened a new log
I20260808 06:59:34.195150 10706 ts_tablet_manager.cc:1397] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:34.195399 10707 tablet_bootstrap.cc:654] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:34.195399 10703 raft_consensus.cc:348] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.195568 10703 raft_consensus.cc:374] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:34.195611 10703 raft_consensus.cc:729] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526e76d4ea62488384543c3b398e060d, State: Initialized, Role: FOLLOWER
I20260808 06:59:34.195748 10703 consensus_queue.cc:260] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.196045 10703 ts_tablet_manager.cc:1428] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 06:59:34.196058 10707 log.cc:824] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:34.196165 10679 heartbeater.cc:503] Master 127.21.28.126:36283 was elected leader, sending a full tablet report...
I20260808 06:59:34.196811 10707 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: No bootstrap required, opened a new log
I20260808 06:59:34.196925 10707 ts_tablet_manager.cc:1397] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:34.197188 10706 raft_consensus.cc:348] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.197337 10706 raft_consensus.cc:374] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:34.197381 10706 raft_consensus.cc:729] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083b9226e90b4e72a1621b1a4603e586, State: Initialized, Role: FOLLOWER
I20260808 06:59:34.197484 10706 consensus_queue.cc:260] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.197734 10706 ts_tablet_manager.cc:1428] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 06:59:34.197760 10467 heartbeater.cc:503] Master 127.21.28.126:36283 was elected leader, sending a full tablet report...
I20260808 06:59:34.198750 10707 raft_consensus.cc:348] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.198868 10707 raft_consensus.cc:374] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:34.198886 10707 raft_consensus.cc:729] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138de8155d4142f3acf3c215f4de8b12, State: Initialized, Role: FOLLOWER
I20260808 06:59:34.198961 10707 consensus_queue.cc:260] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.199203 10707 ts_tablet_manager.cc:1428] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:34.199235 10258 heartbeater.cc:503] Master 127.21.28.126:36283 was elected leader, sending a full tablet report...
I20260808 06:59:34.208721 10712 raft_consensus.cc:482] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:34.208796 10712 raft_consensus.cc:504] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.209110 10712 leader_election.cc:288] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 138de8155d4142f3acf3c215f4de8b12 (127.21.28.65:33111), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
I20260808 06:59:34.211486 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.209230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35863 (local address 127.21.28.66:57317)
0808 06:59:34.209280 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.209292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.209327 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.209451 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.209452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.209453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.209508 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.209509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.210067 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.210068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.210761 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.210763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.211254 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.211255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.211326 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.211328 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.211379 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.211419 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 06:59:34.211483 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.209290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:57317 (local address 127.21.28.67:35863)
0808 06:59:34.209324 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.209325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.209333 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:59:34.209382 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.209383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.209391 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.209435 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.209558 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.210006 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.210842 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.211192 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.211250 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.211259 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.211389 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.211422 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:59:34.211621 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.209299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:55015 (local address 127.21.28.65:33111)
0808 06:59:34.209335 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.209337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.209437 (+ 100us) server_negotiation.cc:407] Connection header received
0808 06:59:34.209503 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.209504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.209512 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.209555 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.209669 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.210114 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.210980 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.211339 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.211397 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.211405 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.211533 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.211563 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:59:34.211634 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.209233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:33111 (local address 127.21.28.66:55015)
0808 06:59:34.209371 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.209387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.209448 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.209556 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.209557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.209559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.209618 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.209619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.210182 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.210183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.210907 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.210909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.211410 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.211411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.211476 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.211479 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.211523 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.211560 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":11,"thread_start_us":72,"threads_started":1}
I20260808 06:59:34.211789 10604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a4cb36bc8f4178adf1e9c428aa793e" candidate_uuid: "083b9226e90b4e72a1621b1a4603e586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526e76d4ea62488384543c3b398e060d" is_pre_election: true
I20260808 06:59:34.211915 10604 raft_consensus.cc:2454] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083b9226e90b4e72a1621b1a4603e586 in term 0.
I20260808 06:59:34.212119 10301 leader_election.cc:302] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083b9226e90b4e72a1621b1a4603e586, 526e76d4ea62488384543c3b398e060d; no voters:
I20260808 06:59:34.212239 10712 raft_consensus.cc:2781] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:59:34.212296 10712 raft_consensus.cc:482] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:59:34.212253 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a4cb36bc8f4178adf1e9c428aa793e" candidate_uuid: "083b9226e90b4e72a1621b1a4603e586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138de8155d4142f3acf3c215f4de8b12" is_pre_election: true
I20260808 06:59:34.212321 10712 raft_consensus.cc:3037] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:34.212378 10185 raft_consensus.cc:2454] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083b9226e90b4e72a1621b1a4603e586 in term 0.
I20260808 06:59:34.212988 10712 raft_consensus.cc:504] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.213097 10712 leader_election.cc:288] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [CANDIDATE]: Term 1 election: Requested vote from peers 138de8155d4142f3acf3c215f4de8b12 (127.21.28.65:33111), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
I20260808 06:59:34.213217 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a4cb36bc8f4178adf1e9c428aa793e" candidate_uuid: "083b9226e90b4e72a1621b1a4603e586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138de8155d4142f3acf3c215f4de8b12"
I20260808 06:59:34.213286 10185 raft_consensus.cc:3037] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:34.213281 10604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a4cb36bc8f4178adf1e9c428aa793e" candidate_uuid: "083b9226e90b4e72a1621b1a4603e586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526e76d4ea62488384543c3b398e060d"
I20260808 06:59:34.213352 10604 raft_consensus.cc:3037] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:34.213939 10185 raft_consensus.cc:2454] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083b9226e90b4e72a1621b1a4603e586 in term 1.
I20260808 06:59:34.213996 10604 raft_consensus.cc:2454] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083b9226e90b4e72a1621b1a4603e586 in term 1.
I20260808 06:59:34.214099 10303 leader_election.cc:302] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083b9226e90b4e72a1621b1a4603e586, 138de8155d4142f3acf3c215f4de8b12; no voters:
I20260808 06:59:34.214205 10712 raft_consensus.cc:2781] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:34.214334 10712 raft_consensus.cc:686] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 1 LEADER]: Becoming Leader. State: Replica: 083b9226e90b4e72a1621b1a4603e586, State: Running, Role: LEADER
I20260808 06:59:34.214414 10712 consensus_queue.cc:237] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.215130 9947 catalog_manager.cc:5495] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66). New cstate: current_term: 1 leader_uuid: "083b9226e90b4e72a1621b1a4603e586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } health_report { overall_health: HEALTHY } } }
I20260808 06:59:34.223627 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.221655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.21.28.66:40403)
0808 06:59:34.221694 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.221695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.221713 (+ 18us) server_negotiation.cc:407] Connection header received
0808 06:59:34.221737 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.221738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.221752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.221797 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:59:34.221898 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.222323 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.223063 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.223231 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.223270 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.223382 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:59:34.223462 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:59:34.223513 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.223540 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.223553 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 06:59:34.225128 10604 raft_consensus.cc:1261] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 1 FOLLOWER]: Refusing update from remote peer 083b9226e90b4e72a1621b1a4603e586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:34.225129 10185 raft_consensus.cc:1261] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Refusing update from remote peer 083b9226e90b4e72a1621b1a4603e586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:34.225426 10712 consensus_queue.cc:1035] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:34.225558 10716 consensus_queue.cc:1035] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:34.227702 10719 mvcc.cc:205] Tried to move back new op lower bound from 7316162044824027136 to 7316162044782792704. Current Snapshot: MvccSnapshot[applied={T|T < 7316162044824027136}]
I20260808 06:59:34.227958 10720 mvcc.cc:205] Tried to move back new op lower bound from 7316162044824027136 to 7316162044782792704. Current Snapshot: MvccSnapshot[applied={T|T < 7316162044824027136}]
I20260808 06:59:34.227958 10721 mvcc.cc:205] Tried to move back new op lower bound from 7316162044824027136 to 7316162044782792704. Current Snapshot: MvccSnapshot[applied={T|T < 7316162044824027136}]
I20260808 06:59:34.229976 9946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:59:34.230072 9946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:59:34.230773 9946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Sending DeleteTablet for 3 replicas of tablet f8a4cb36bc8f4178adf1e9c428aa793e
I20260808 06:59:34.231070 10367 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a4cb36bc8f4178adf1e9c428aa793e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:34 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:36596
I20260808 06:59:34.231087 10577 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a4cb36bc8f4178adf1e9c428aa793e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:34 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:34796
I20260808 06:59:34.231213 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 10054
I20260808 06:59:34.231341 10151 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a4cb36bc8f4178adf1e9c428aa793e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:34 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:40660
I20260808 06:59:34.239806 10736 tablet_replica.cc:330] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: stopping tablet replica
I20260808 06:59:34.239967 10736 raft_consensus.cc:2229] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:59:34.240139 10736 raft_consensus.cc:2258] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:34.240621 10736 ts_tablet_manager.cc:1905] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:34.241998 10738 tablet_replica.cc:330] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: stopping tablet replica
I20260808 06:59:34.242144 10738 raft_consensus.cc:2229] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:59:34.242254 10738 raft_consensus.cc:2258] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:34.242648 10738 ts_tablet_manager.cc:1905] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:34.243818 10736 ts_tablet_manager.cc:1918] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:34.243896 10736 log.cc:1196] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/f8a4cb36bc8f4178adf1e9c428aa793e
I20260808 06:59:34.244144 10736 ts_tablet_manager.cc:1939] T f8a4cb36bc8f4178adf1e9c428aa793e P 083b9226e90b4e72a1621b1a4603e586: Deleting consensus metadata
I20260808 06:59:34.244215 10738 ts_tablet_manager.cc:1918] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:34.244284 10738 log.cc:1196] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/f8a4cb36bc8f4178adf1e9c428aa793e
I20260808 06:59:34.244534 10738 ts_tablet_manager.cc:1939] T f8a4cb36bc8f4178adf1e9c428aa793e P 526e76d4ea62488384543c3b398e060d: Deleting consensus metadata
I20260808 06:59:34.244565 9926 catalog_manager.cc:4841] TS 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403): tablet f8a4cb36bc8f4178adf1e9c428aa793e (table test-table [id=c3e005b049d8453a823844c13bfa09fc]) successfully deleted
I20260808 06:59:34.244910 9927 catalog_manager.cc:4841] TS 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863): tablet f8a4cb36bc8f4178adf1e9c428aa793e (table test-table [id=c3e005b049d8453a823844c13bfa09fc]) successfully deleted
W20260808 06:59:34.247284 9928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:33111 (error 108)
I20260808 06:59:34.247781 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.247629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:33111 (local address 127.0.0.1:40672)
0808 06:59:34.247706 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.247746 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 06:59:34.247851 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 10265
W20260808 06:59:34.250711 9928 catalog_manager.cc:4584] TS 138de8155d4142f3acf3c215f4de8b12 (127.21.28.65:33111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8a4cb36bc8f4178adf1e9c428aa793e: Network error: Client connection negotiation failed: client connection to 127.21.28.65:33111: connect: Connection refused (error 111)
I20260808 06:59:34.256579 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 10473
I20260808 06:59:34.266446 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 9901
I20260808 06:59:34.277954 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36283
--webserver_interface=127.21.28.126
--webserver_port=45913
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:34.357054 10740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:34.357249 10740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:34.357270 10740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:34.358369 10740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:34.358417 10740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:34.358444 10740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:34.358469 10740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:34.358489 10740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:34.359962 10740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=45913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:34.360185 10740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:34.360401 10740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:34.368973 10749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.368944 10751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.369144 10753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.369211 10740 server_base.cc:1034] running on GCE node
I20260808 06:59:34.369431 10740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:34.369695 10740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:34.370824 10740 hybrid_clock.cc:648] HybridClock initialized: now 1786172374370806 us; error 35 us; skew 500 ppm
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:34.377444 10740 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:59:34.377873 10740 webserver.cc:460] Webserver started at http://127.21.28.126:45913/ using document root <none> and password file <none>
I20260808 06:59:34.378100 10740 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:34.378158 10740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:34.379446 10740 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:34.380059 10764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.380199 10740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.380280 10740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "14cd4072046a40ebbdad734ed393884b"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "a60aab80b08924ec351c4ee7d54f80bd"
server_key_iv: "202ccd74e685d6600863feb436b97271"
server_key_version: "encryptionkey@0"
I20260808 06:59:34.380661 10740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:34.396070 10740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:34.397640 10740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:34.397759 10740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:34.401549 10740 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:36283
I20260808 06:59:34.401582 10838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:36283 every 8 connection(s)
I20260808 06:59:34.401928 10740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:59:34.402503 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 10740
I20260808 06:59:34.402760 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:33111
--local_ip_for_outbound_sockets=127.21.28.65
--tserver_master_addrs=127.21.28.126:36283
--webserver_port=46489
--webserver_interface=127.21.28.65
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:59:34.403613 10840 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:59:34.404304 10840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Bootstrap starting.
I20260808 06:59:34.412441 10840 log.cc:824] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:34.414224 10840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:59:34.414528 10840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Bootstrap complete.
I20260808 06:59:34.416702 10840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:34.416920 10840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cd4072046a40ebbdad734ed393884b, State: Initialized, Role: FOLLOWER
I20260808 06:59:34.417071 10840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:34.417145 10840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:59:34.417198 10840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:59:34.417246 10840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:59:34.418220 10840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:34.418315 10840 leader_election.cc:302] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14cd4072046a40ebbdad734ed393884b; no voters:
I20260808 06:59:34.418476 10840 leader_election.cc:288] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 06:59:34.418524 10844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:59:34.418637 10844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 2 LEADER]: Becoming Leader. State: Replica: 14cd4072046a40ebbdad734ed393884b, State: Running, Role: LEADER
I20260808 06:59:34.418747 10840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:34.418769 10844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:34.419011 10844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cd4072046a40ebbdad734ed393884b. Latest consensus state: current_term: 2 leader_uuid: "14cd4072046a40ebbdad734ed393884b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } } }
I20260808 06:59:34.419073 10844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:34.419167 10845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "14cd4072046a40ebbdad734ed393884b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } } }
I20260808 06:59:34.419257 10855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:59:34.419297 10845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:34.420097 10855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:59:34.420217 10855 catalog_manager.cc:1259] Loaded cluster ID: 68b7936c95c14d73847ef045fb65920b
I20260808 06:59:34.420423 10855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:59:34.420713 10855 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 06:59:34.423363 10855 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 06:59:34.425034 10855 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:59:34.425204 10855 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Loaded TSK: 0
I20260808 06:59:34.425410 10855 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:59:34.501040 10842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:34.501240 10842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:34.501271 10842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:34.502369 10842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:34.502408 10842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:34.502441 10842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:59:34.503916 10842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:33111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=46489
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:34.504132 10842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:34.504330 10842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:34.509071 10877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.509109 10878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.509076 10881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.509513 10842 server_base.cc:1034] running on GCE node
I20260808 06:59:34.509737 10842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:34.510003 10842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:34.511190 10842 hybrid_clock.cc:648] HybridClock initialized: now 1786172374511177 us; error 27 us; skew 500 ppm
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:34.514271 10842 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:59:34.514815 10842 webserver.cc:460] Webserver started at http://127.21.28.65:46489/ using document root <none> and password file <none>
I20260808 06:59:34.515070 10842 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:34.515154 10842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:34.516960 10842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:34.517699 10890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.517911 10842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:34.517998 10842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "138de8155d4142f3acf3c215f4de8b12"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "4973b456fc69cda36e9d45feff07b42d"
server_key_iv: "be64b624ad8f2f11def2287cb958fba2"
server_key_version: "encryptionkey@0"
I20260808 06:59:34.518297 10842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:34.527132 10842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:34.529065 10842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:34.529196 10842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:34.529381 10842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:34.529788 10897 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 06:59:34.531581 10842 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 06:59:34.531626 10842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 06:59:34.531656 10842 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 06:59:34.532269 10842 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 06:59:34.532303 10842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 06:59:34.532397 10897 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Bootstrap starting.
I20260808 06:59:34.538928 10842 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:33111
I20260808 06:59:34.539266 10842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:59:34.543622 10897 log.cc:824] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:34.543955 11056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:33111 every 8 connection(s)
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:34.546339 10897 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:59:34.546662 10897 tablet_bootstrap.cc:492] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Bootstrap complete.
I20260808 06:59:34.548024 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 10842
I20260808 06:59:34.548322 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:40403
--local_ip_for_outbound_sockets=127.21.28.66
--tserver_master_addrs=127.21.28.126:36283
--webserver_port=35841
--webserver_interface=127.21.28.66
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:59:34.549405 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.539965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:44867)
0808 06:59:34.540122 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.540161 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.540621 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.540996 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.541000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.541256 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:34.541432 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.541437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.542533 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.542534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.543284 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.543286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.543987 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.543988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.544134 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.544661 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:34.544669 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:34.546022 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:34.547846 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.547853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.547855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.548061 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.548308 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.548309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.548309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.548433 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.548809 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:34.548816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:34.548939 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.549024 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.549192 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260808 06:59:34.549404 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.540101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:44867 (local address 127.21.28.126:36283)
0808 06:59:34.540270 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.540277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.540324 (+ 47us) server_negotiation.cc:407] Connection header received
0808 06:59:34.540809 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.540813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.540853 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.540961 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:34.541539 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.542421 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.543363 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.543872 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.544397 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.546134 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:34.546138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:34.546139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:34.546150 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:34.547779 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.548149 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.548152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.548152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.548242 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.548508 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.548509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.548509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.548738 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:34.548810 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.549094 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.549211 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260808 06:59:34.547791 10897 ts_tablet_manager.cc:1397] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.015s
I20260808 06:59:34.550518 11057 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36283
I20260808 06:59:34.550604 11057 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:34.550760 11057 heartbeater.cc:511] Master 127.21.28.126:36283 requested a full tablet report, sending...
I20260808 06:59:34.551430 10789 ts_manager.cc:194] Registered new tserver with Master: 138de8155d4142f3acf3c215f4de8b12 (127.21.28.65:33111)
I20260808 06:59:34.551229 10897 raft_consensus.cc:348] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.551548 10897 raft_consensus.cc:729] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138de8155d4142f3acf3c215f4de8b12, State: Initialized, Role: FOLLOWER
I20260808 06:59:34.551713 10897 consensus_queue.cc:260] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } }
I20260808 06:59:34.551963 10897 ts_tablet_manager.cc:1428] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:34.552551 10789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:44867
I20260808 06:59:34.553335 11057 heartbeater.cc:503] Master 127.21.28.126:36283 was elected leader, sending a full tablet report...
I20260808 06:59:34.556986 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.552554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.21.28.65:33111)
0808 06:59:34.553127 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.553131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.553140 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:59:34.553169 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.553171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.553221 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.553306 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.553487 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.554457 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.555925 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.556588 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.556661 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.556708 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.556892 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.556927 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":72,"threads_started":1}
I20260808 06:59:34.557009 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.552330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:33111 (local address 127.0.0.1:40688)
0808 06:59:34.552922 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.552955 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.553011 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.553334 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.553337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.553339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.553421 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.553423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.554542 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.554543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.555849 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.555852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.556673 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.556674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.556776 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.556802 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.556877 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.556930 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":77,"threads_started":1}
I20260808 06:59:34.557557 10956 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a4cb36bc8f4178adf1e9c428aa793e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:34 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:40688
I20260808 06:59:34.558077 11074 tablet_replica.cc:330] stopping tablet replica
I20260808 06:59:34.558183 11074 raft_consensus.cc:2229] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:59:34.558249 11074 raft_consensus.cc:2258] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:34.558538 11074 ts_tablet_manager.cc:1905] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:34.560827 11074 ts_tablet_manager.cc:1918] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:34.560904 11074 log.cc:1196] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/f8a4cb36bc8f4178adf1e9c428aa793e
I20260808 06:59:34.561132 11074 ts_tablet_manager.cc:1939] T f8a4cb36bc8f4178adf1e9c428aa793e P 138de8155d4142f3acf3c215f4de8b12: Deleting consensus metadata
I20260808 06:59:34.561619 10768 catalog_manager.cc:4841] TS 138de8155d4142f3acf3c215f4de8b12 (127.21.28.65:33111): tablet f8a4cb36bc8f4178adf1e9c428aa793e (table test-table [id=c3e005b049d8453a823844c13bfa09fc]) successfully deleted
W20260808 06:59:34.632231 11064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:34.632439 11064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:34.632472 11064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:34.633598 11064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:34.633637 11064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:34.633671 11064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:59:34.635068 11064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:40403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=35841
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:34.635299 11064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:34.635504 11064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:34.640239 11081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.640343 11064 server_base.cc:1034] running on GCE node
W20260808 06:59:34.640232 11082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.640233 11084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.640657 11064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:34.640888 11064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:34.642031 11064 hybrid_clock.cc:648] HybridClock initialized: now 1786172374642012 us; error 33 us; skew 500 ppm
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:34.644346 11064 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:59:34.644755 11064 webserver.cc:460] Webserver started at http://127.21.28.66:35841/ using document root <none> and password file <none>
I20260808 06:59:34.644963 11064 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:34.645027 11064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:34.646306 11064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:59:34.646852 11091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.647012 11064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 06:59:34.647095 11064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "083b9226e90b4e72a1621b1a4603e586"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "fe5432b386f0c00cd3ff217c0e19d907"
server_key_iv: "5703cca9e971c2045a29997ee0b41a79"
server_key_version: "encryptionkey@0"
I20260808 06:59:34.647401 11064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:34.656610 11064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:34.658223 11064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:34.658347 11064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:34.658542 11064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:34.658806 11064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:34.658838 11064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.658864 11064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:34.658888 11064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.664574 11064 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40403
I20260808 06:59:34.664613 11262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40403 every 8 connection(s)
I20260808 06:59:34.664915 11064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:59:34.665791 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 11064
I20260808 06:59:34.666079 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:35863
--local_ip_for_outbound_sockets=127.21.28.67
--tserver_master_addrs=127.21.28.126:36283
--webserver_port=44879
--webserver_interface=127.21.28.67
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:34.674820 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.665534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:51701 (local address 127.21.28.126:36283)
0808 06:59:34.665612 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.665614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.665840 (+ 226us) server_negotiation.cc:407] Connection header received
0808 06:59:34.666465 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.666466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.666478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.666530 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:34.667693 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.668459 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.669417 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.669714 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.669786 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.672151 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:34.672153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:34.672154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:34.672162 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:34.673598 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.673975 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.673977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.673977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.674037 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.674234 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.674235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.674235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.674364 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:34.674435 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.674636 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.674722 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 06:59:34.675040 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.665515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:51701)
0808 06:59:34.665681 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.665725 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.666370 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.666599 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.666603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.667323 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:34.667586 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.667592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.668559 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.668560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.669337 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.669340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.669805 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.669806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.669903 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.670379 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:34.670388 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:34.672068 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:34.673689 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.673695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.673696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.673907 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.674097 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.674098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.674098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.674184 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.674420 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:34.674422 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:34.674524 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.674596 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.674750 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260808 06:59:34.675472 11263 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36283
I20260808 06:59:34.675554 11263 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:34.675748 11263 heartbeater.cc:511] Master 127.21.28.126:36283 requested a full tablet report, sending...
I20260808 06:59:34.676116 10789 ts_manager.cc:194] Registered new tserver with Master: 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403)
I20260808 06:59:34.676477 10789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:51701
W20260808 06:59:34.748759 11270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:34.748950 11270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:34.748982 11270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:34.750154 11270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:34.750195 11270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:34.750229 11270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:59:34.751780 11270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:35863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=44879
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:34.752008 11270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:34.752211 11270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:34.756949 11282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.756966 11277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.756999 11270 server_base.cc:1034] running on GCE node
W20260808 06:59:34.756975 11279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.757331 11270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:34.757557 11270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:34.758690 11270 hybrid_clock.cc:648] HybridClock initialized: now 1786172374758674 us; error 35 us; skew 500 ppm
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:34.761014 11270 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:59:34.761405 11270 webserver.cc:460] Webserver started at http://127.21.28.67:44879/ using document root <none> and password file <none>
I20260808 06:59:34.761606 11270 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:34.761664 11270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:34.762845 11270 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:59:34.763334 11296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.763485 11270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.763572 11270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "526e76d4ea62488384543c3b398e060d"
format_stamp: "Formatted at 2026-08-08 06:59:34 on dist-test-slave-nj21"
server_key: "34ff11ef621b8f727a8df2b1cebf0dbf"
server_key_iv: "349e9647a61de739055e9e5b25d8e424"
server_key_version: "encryptionkey@0"
I20260808 06:59:34.763890 11270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:34.778198 11270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:34.779901 11270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:34.780004 11270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:34.780169 11270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:34.780405 11270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:34.780435 11270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.780454 11270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:34.780465 11270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.786285 11270 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:35863
I20260808 06:59:34.786343 11477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:35863 every 8 connection(s)
I20260808 06:59:34.786626 11270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:59:34.791879 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 11270
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:34.795276 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.787277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:54997 (local address 127.21.28.126:36283)
0808 06:59:34.787321 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.787323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.787524 (+ 201us) server_negotiation.cc:407] Connection header received
0808 06:59:34.788032 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.788033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.788046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.788094 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:34.788974 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.789475 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.790430 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.790620 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.790676 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.792677 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:34.792678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:34.792678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:34.792685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:34.794056 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.794414 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.794415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.794416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.794480 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.794678 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.794678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.794679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.794797 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:34.794858 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.795123 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.795189 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 06:59:34.795468 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.787249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:54997)
0808 06:59:34.787414 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.787447 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.787932 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.788170 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.788174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.788696 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:34.788872 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.788887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.789558 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.789558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.790341 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.790343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.790686 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.790687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.790815 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.791208 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:34.791215 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:34.792600 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:34.794130 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.794135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.794137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.794344 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.794541 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:34.794541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:34.794541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:34.794630 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:34.794853 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:34.794859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:34.795001 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.795102 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.795256 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260808 06:59:34.795903 11478 heartbeater.cc:347] Connected to a master server at 127.21.28.126:36283
I20260808 06:59:34.796000 11478 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:34.796181 11478 heartbeater.cc:511] Master 127.21.28.126:36283 requested a full tablet report, sending...
I20260808 06:59:34.796530 10789 ts_manager.cc:194] Registered new tserver with Master: 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
I20260808 06:59:34.796891 10789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:54997
I20260808 06:59:34.798856 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.792252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.21.28.126:36283)
0808 06:59:34.792496 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.792498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.792503 (+ 5us) server_negotiation.cc:407] Connection header received
0808 06:59:34.792520 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.792520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.792529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.792576 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:34.793296 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.793852 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.794860 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.795160 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.795220 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.796596 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:34.796597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:34.796597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:34.796603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:34.797905 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.798197 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.798198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.798198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.798247 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:34.798408 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:34.798409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:34.798409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:34.798505 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:34.798661 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.798696 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.798768 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":58,"threads_started":1}
I20260808 06:59:34.799420 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:59:34.801640 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.799735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.21.28.126:36283)
0808 06:59:34.799781 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.799783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.799853 (+ 70us) server_negotiation.cc:407] Connection header received
0808 06:59:34.799909 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.799910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.799937 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.799987 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:59:34.800060 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.800484 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.801141 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.801310 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.801349 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.801431 (+ 82us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:59:34.801507 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:59:34.801554 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.801570 (+ 16us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.801585 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 06:59:34.801904 10789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:59:34.803424 10789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:59:34.805310 10956 tablet_service.cc:1467] Processing CreateTablet for tablet c524e48b474847a89169473e2e38c112 (DEFAULT_TABLE table=test-table [id=6e23b26f64d74c1eb28eaad3bdc6d469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:34.805509 10956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c524e48b474847a89169473e2e38c112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:34.807013 10897 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Bootstrap starting.
I20260808 06:59:34.807305 10897 tablet_bootstrap.cc:654] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:34.807783 10897 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: No bootstrap required, opened a new log
I20260808 06:59:34.807838 10897 ts_tablet_manager.cc:1397] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 06:59:34.807943 10897 raft_consensus.cc:348] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.808007 10897 raft_consensus.cc:374] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:34.808041 10897 raft_consensus.cc:729] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138de8155d4142f3acf3c215f4de8b12, State: Initialized, Role: FOLLOWER
I20260808 06:59:34.808082 10897 consensus_queue.cc:260] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.808159 10897 ts_tablet_manager.cc:1428] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.808727 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.805050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40403 (local address 127.0.0.1:46228)
0808 06:59:34.805094 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.805103 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.805134 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.805510 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.805511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.805513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.805547 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.805548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.806549 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.806551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.807592 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.807594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.808497 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.808498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.808561 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.808564 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.808622 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.808664 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 06:59:34.808770 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.805156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.21.28.66:40403)
0808 06:59:34.805346 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.805351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.805361 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:59:34.805414 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.805416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.805434 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.805515 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.805588 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.806467 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.807664 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.808421 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.808494 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.808555 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.808638 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.808685 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260808 06:59:34.808887 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.805079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35863 (local address 127.0.0.1:60050)
0808 06:59:34.805225 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.805233 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.805281 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.805569 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.805570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.805572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.805614 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.805616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.806767 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.806768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.807834 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.807837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.808705 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.808706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.808763 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.808766 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.808794 (+ 28us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.808829 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260808 06:59:34.809185 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.805263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.21.28.67:35863)
0808 06:59:34.805412 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.805416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.805425 (+ 9us) server_negotiation.cc:407] Connection header received
0808 06:59:34.805455 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.805456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.805469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.805679 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.805723 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.806662 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.807920 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.808625 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.808987 (+ 362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.809055 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.809079 (+ 24us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.809116 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260808 06:59:34.809504 11166 tablet_service.cc:1467] Processing CreateTablet for tablet c524e48b474847a89169473e2e38c112 (DEFAULT_TABLE table=test-table [id=6e23b26f64d74c1eb28eaad3bdc6d469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:34.809684 11377 tablet_service.cc:1467] Processing CreateTablet for tablet c524e48b474847a89169473e2e38c112 (DEFAULT_TABLE table=test-table [id=6e23b26f64d74c1eb28eaad3bdc6d469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:34.809798 11166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c524e48b474847a89169473e2e38c112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:34.809967 11377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c524e48b474847a89169473e2e38c112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:34.812362 11492 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: Bootstrap starting.
I20260808 06:59:34.812654 11493 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: Bootstrap starting.
I20260808 06:59:34.812969 11492 tablet_bootstrap.cc:654] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:34.813190 11493 tablet_bootstrap.cc:654] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:34.813465 11492 log.cc:824] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:34.813651 11067 raft_consensus.cc:482] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:34.813737 11067 raft_consensus.cc:504] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.813771 11493 log.cc:824] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:34.813947 11067 leader_election.cc:288] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
I20260808 06:59:34.814482 11493 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: No bootstrap required, opened a new log
I20260808 06:59:34.814483 11492 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: No bootstrap required, opened a new log
I20260808 06:59:34.814555 11492 ts_tablet_manager.cc:1397] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:59:34.814556 11493 ts_tablet_manager.cc:1397] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:59:34.816164 11492 raft_consensus.cc:348] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.816205 11493 raft_consensus.cc:348] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.816319 11493 raft_consensus.cc:374] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:34.816319 11492 raft_consensus.cc:374] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:34.816349 11493 raft_consensus.cc:729] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526e76d4ea62488384543c3b398e060d, State: Initialized, Role: FOLLOWER
I20260808 06:59:34.816351 11492 raft_consensus.cc:729] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083b9226e90b4e72a1621b1a4603e586, State: Initialized, Role: FOLLOWER
I20260808 06:59:34.816465 11493 consensus_queue.cc:260] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.816478 11492 consensus_queue.cc:260] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.816740 11493 ts_tablet_manager.cc:1428] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:34.816893 11478 heartbeater.cc:503] Master 127.21.28.126:36283 was elected leader, sending a full tablet report...
I20260808 06:59:34.816754 11492 ts_tablet_manager.cc:1428] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:34.817440 11263 heartbeater.cc:503] Master 127.21.28.126:36283 was elected leader, sending a full tablet report...
I20260808 06:59:34.817648 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.814071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40403 (local address 127.21.28.65:55031)
0808 06:59:34.814246 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.814256 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.814316 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.814472 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.814473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.814475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.814538 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.814540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.815359 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.815360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.816513 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.816516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.817373 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.817374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.817466 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.817470 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.817519 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.817562 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260808 06:59:34.817936 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.814158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:55031 (local address 127.21.28.66:40403)
0808 06:59:34.814192 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.814194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.814304 (+ 110us) server_negotiation.cc:407] Connection header received
0808 06:59:34.814362 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.814363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.814371 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.814426 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.814615 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.815267 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.816617 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.817246 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.817317 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.817324 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.817841 (+ 517us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.817881 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 06:59:34.818207 11192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c524e48b474847a89169473e2e38c112" candidate_uuid: "138de8155d4142f3acf3c215f4de8b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083b9226e90b4e72a1621b1a4603e586" is_pre_election: true
I20260808 06:59:34.818320 11192 raft_consensus.cc:2454] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138de8155d4142f3acf3c215f4de8b12 in term 0.
I20260808 06:59:34.818496 10892 leader_election.cc:302] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083b9226e90b4e72a1621b1a4603e586, 138de8155d4142f3acf3c215f4de8b12; no voters:
I20260808 06:59:34.818614 11067 raft_consensus.cc:2781] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:59:34.818688 11067 raft_consensus.cc:482] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:59:34.818720 11067 raft_consensus.cc:3037] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:34.819285 11067 raft_consensus.cc:504] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.819386 11067 leader_election.cc:288] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 1 election: Requested vote from peers 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
I20260808 06:59:34.819547 11192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c524e48b474847a89169473e2e38c112" candidate_uuid: "138de8155d4142f3acf3c215f4de8b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083b9226e90b4e72a1621b1a4603e586"
I20260808 06:59:34.819624 11192 raft_consensus.cc:3037] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:34.820430 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.814079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35863 (local address 127.21.28.65:56411)
0808 06:59:34.814410 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.814418 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:34.814453 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:34.816609 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:34.816609 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:34.816611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:34.816678 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.816679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.817612 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.817613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.819536 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:34.819539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.820235 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.820236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:34.820298 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.820302 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:59:34.820345 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 06:59:34.820371 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":8,"thread_start_us":79,"threads_started":1}
I20260808 06:59:34.820430 11192 raft_consensus.cc:2454] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138de8155d4142f3acf3c215f4de8b12 in term 1.
I20260808 06:59:34.820459 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.814244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:56411 (local address 127.21.28.67:35863)
0808 06:59:34.815760 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.815762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.815770 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:59:34.815787 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.815787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.815796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.815849 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:34.816761 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.817420 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.819612 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.820158 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.820238 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.820245 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.820363 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.820406 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492}
I20260808 06:59:34.820603 10892 leader_election.cc:302] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083b9226e90b4e72a1621b1a4603e586, 138de8155d4142f3acf3c215f4de8b12; no voters:
I20260808 06:59:34.820684 11067 raft_consensus.cc:2781] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:34.820712 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c524e48b474847a89169473e2e38c112" candidate_uuid: "138de8155d4142f3acf3c215f4de8b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526e76d4ea62488384543c3b398e060d" is_pre_election: true
I20260808 06:59:34.820817 11067 raft_consensus.cc:686] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 LEADER]: Becoming Leader. State: Replica: 138de8155d4142f3acf3c215f4de8b12, State: Running, Role: LEADER
I20260808 06:59:34.820845 11401 raft_consensus.cc:2454] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138de8155d4142f3acf3c215f4de8b12 in term 0.
I20260808 06:59:34.820883 11067 consensus_queue.cc:237] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:34.820972 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c524e48b474847a89169473e2e38c112" candidate_uuid: "138de8155d4142f3acf3c215f4de8b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526e76d4ea62488384543c3b398e060d"
I20260808 06:59:34.821033 11400 raft_consensus.cc:3037] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:34.821566 10789 catalog_manager.cc:5495] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 138de8155d4142f3acf3c215f4de8b12 (127.21.28.65). New cstate: current_term: 1 leader_uuid: "138de8155d4142f3acf3c215f4de8b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } health_report { overall_health: HEALTHY } } }
I20260808 06:59:34.821741 11400 raft_consensus.cc:2454] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138de8155d4142f3acf3c215f4de8b12 in term 1.
I20260808 06:59:34.830539 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.828499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.21.28.65:33111)
0808 06:59:34.828553 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:34.828555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:34.828562 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:59:34.828590 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:34.828591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:34.828607 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:34.828675 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:59:34.828798 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.829310 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.830024 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:34.830192 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:34.830232 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:34.830327 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:59:34.830410 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:59:34.830451 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:34.830469 (+ 18us) server_negotiation.cc:299] Negotiation successful
0808 06:59:34.830483 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:59:34.831779 11192 raft_consensus.cc:1261] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 1 FOLLOWER]: Refusing update from remote peer 138de8155d4142f3acf3c215f4de8b12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:34.831791 11400 raft_consensus.cc:1261] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 1 FOLLOWER]: Refusing update from remote peer 138de8155d4142f3acf3c215f4de8b12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:34.832073 11502 consensus_queue.cc:1035] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:34.832185 11067 consensus_queue.cc:1035] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:34.834408 11509 mvcc.cc:205] Tried to move back new op lower bound from 7316162047308861440 to 7316162047266836480. Current Snapshot: MvccSnapshot[applied={T|T < 7316162047308861440}]
I20260808 06:59:34.835052 11508 mvcc.cc:205] Tried to move back new op lower bound from 7316162047308861440 to 7316162047266836480. Current Snapshot: MvccSnapshot[applied={T|T < 7316162047308861440}]
I20260808 06:59:34.836133 10785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:59:34.836225 10785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:59:34.836992 10785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Sending DeleteTablet for 3 replicas of tablet c524e48b474847a89169473e2e38c112
I20260808 06:59:34.837306 11166 tablet_service.cc:1514] Processing DeleteTablet for tablet c524e48b474847a89169473e2e38c112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:34 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:46228
I20260808 06:59:34.837325 11377 tablet_service.cc:1514] Processing DeleteTablet for tablet c524e48b474847a89169473e2e38c112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:34 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:60050
I20260808 06:59:34.837508 11524 tablet_replica.cc:330] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: stopping tablet replica
I20260808 06:59:34.837590 11525 tablet_replica.cc:330] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: stopping tablet replica
I20260808 06:59:34.837606 11524 raft_consensus.cc:2229] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:59:34.837699 11524 raft_consensus.cc:2258] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:34.837702 10956 tablet_service.cc:1514] Processing DeleteTablet for tablet c524e48b474847a89169473e2e38c112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:34 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:40688
I20260808 06:59:34.837719 11525 raft_consensus.cc:2229] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:59:34.837834 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 10842
I20260808 06:59:34.837965 11525 raft_consensus.cc:2258] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:34.838140 11524 ts_tablet_manager.cc:1905] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:34.838500 11525 ts_tablet_manager.cc:1905] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:34.839960 11525 ts_tablet_manager.cc:1918] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:34.840021 11525 log.cc:1196] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/c524e48b474847a89169473e2e38c112
I20260808 06:59:34.840233 11525 ts_tablet_manager.cc:1939] T c524e48b474847a89169473e2e38c112 P 526e76d4ea62488384543c3b398e060d: Deleting consensus metadata
I20260808 06:59:34.840606 10767 catalog_manager.cc:4841] TS 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863): tablet c524e48b474847a89169473e2e38c112 (table test-table [id=6e23b26f64d74c1eb28eaad3bdc6d469]) successfully deleted
I20260808 06:59:34.841431 11524 ts_tablet_manager.cc:1918] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:34.841478 11524 log.cc:1196] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/c524e48b474847a89169473e2e38c112
I20260808 06:59:34.841653 11524 ts_tablet_manager.cc:1939] T c524e48b474847a89169473e2e38c112 P 083b9226e90b4e72a1621b1a4603e586: Deleting consensus metadata
I20260808 06:59:34.842092 10766 catalog_manager.cc:4841] TS 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403): tablet c524e48b474847a89169473e2e38c112 (table test-table [id=6e23b26f64d74c1eb28eaad3bdc6d469]) successfully deleted
W20260808 06:59:34.847390 10768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:33111 (error 108)
I20260808 06:59:34.847725 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 11064
I20260808 06:59:34.847875 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:34.847775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:33111 (local address 127.0.0.1:34206)
0808 06:59:34.847820 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:34.847848 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 06:59:34.851070 10768 catalog_manager.cc:4584] TS 138de8155d4142f3acf3c215f4de8b12 (127.21.28.65:33111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c524e48b474847a89169473e2e38c112: Network error: Client connection negotiation failed: client connection to 127.21.28.65:33111: connect: Connection refused (error 111)
I20260808 06:59:34.856400 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 11270
I20260808 06:59:34.867297 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 10740
I20260808 06:59:34.876004 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36283
--webserver_interface=127.21.28.126
--webserver_port=45913
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:34.954174 11528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:34.954362 11528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:34.954383 11528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:34.955485 11528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:34.955524 11528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:34.955538 11528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:34.955549 11528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:34.955561 11528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:34.957031 11528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=45913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:34.957247 11528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:34.957469 11528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:34.961948 11535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.962008 11539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:34.961982 11537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:34.962159 11528 server_base.cc:1034] running on GCE node
I20260808 06:59:34.962366 11528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:34.962625 11528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:34.963719 11528 hybrid_clock.cc:648] HybridClock initialized: now 1786172374963707 us; error 29 us; skew 500 ppm
Aug 08 06:59:34 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172374, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:34.966061 11528 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:59:34.966471 11528 webserver.cc:460] Webserver started at http://127.21.28.126:45913/ using document root <none> and password file <none>
I20260808 06:59:34.966678 11528 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:34.966725 11528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:34.967948 11528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:59:34.968622 11553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:34.968873 11528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:59:34.968968 11528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "14cd4072046a40ebbdad734ed393884b"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "a60aab80b08924ec351c4ee7d54f80bd"
server_key_iv: "202ccd74e685d6600863feb436b97271"
server_key_version: "encryptionkey@0"
I20260808 06:59:34.969297 11528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:34.980913 11528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:34.983814 11528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:34.983999 11528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:34.988196 11528 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:36283
I20260808 06:59:34.988234 11621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:36283 every 8 connection(s)
I20260808 06:59:34.988596 11528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:59:34.990223 11622 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:59:34.990339 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 11528
I20260808 06:59:34.990641 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:33111
--local_ip_for_outbound_sockets=127.21.28.65
--tserver_master_addrs=127.21.28.126:36283
--webserver_port=46489
--webserver_interface=127.21.28.65
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:59:34.991117 11622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Bootstrap starting.
I20260808 06:59:34.999752 11622 log.cc:824] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:35.002115 11622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:59:35.002421 11622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b: Bootstrap complete.
I20260808 06:59:35.004678 11622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:35.004905 11622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cd4072046a40ebbdad734ed393884b, State: Initialized, Role: FOLLOWER
I20260808 06:59:35.005048 11622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:35.005115 11622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:59:35.005143 11622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:59:35.005188 11622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 2 FOLLOWER]: Advancing to term 3
I20260808 06:59:35.006157 11622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:35.006253 11622 leader_election.cc:302] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14cd4072046a40ebbdad734ed393884b; no voters:
I20260808 06:59:35.006417 11622 leader_election.cc:288] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 06:59:35.006464 11628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 3 FOLLOWER]: Leader election won for term 3
I20260808 06:59:35.006577 11628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [term 3 LEADER]: Becoming Leader. State: Replica: 14cd4072046a40ebbdad734ed393884b, State: Running, Role: LEADER
I20260808 06:59:35.006690 11622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:35.006668 11628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } }
I20260808 06:59:35.006965 11631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cd4072046a40ebbdad734ed393884b. Latest consensus state: current_term: 3 leader_uuid: "14cd4072046a40ebbdad734ed393884b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } } }
I20260808 06:59:35.006961 11629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "14cd4072046a40ebbdad734ed393884b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd4072046a40ebbdad734ed393884b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36283 } } }
I20260808 06:59:35.007093 11629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:35.007054 11631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cd4072046a40ebbdad734ed393884b [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:35.007552 11639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:59:35.008385 11639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:59:35.008507 11639 catalog_manager.cc:1259] Loaded cluster ID: 68b7936c95c14d73847ef045fb65920b
I20260808 06:59:35.008702 11639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 06:59:35.008935 11639 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fec79be39ad google::LogMessage::Fail() at ??:0
@ 0x7fec79be688a google::LogMessage::SendToLog() at ??:0
@ 0x7fec79be34de google::LogMessage::Flush() at ??:0
@ 0x7fec79be4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fec81b040f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fec81b0d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fec7bab10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fec7baa6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fec7c90a6db start_thread at ??:0
@ 0x7fec786e161f clone at ??:0
W20260808 06:59:35.087363 11625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:35.087549 11625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:35.087570 11625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:35.088703 11625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:35.088744 11625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:35.088769 11625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:59:35.090157 11625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:33111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=46489
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:35.090363 11625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:35.090570 11625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:35.095194 11678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:35.095189 11681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:35.095532 11625 server_base.cc:1034] running on GCE node
W20260808 06:59:35.095330 11676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:35.095865 11625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:35.096083 11625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:35.097214 11625 hybrid_clock.cc:648] HybridClock initialized: now 1786172375097204 us; error 34 us; skew 500 ppm
Aug 08 06:59:35 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172375, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:35.099769 11625 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:59:35.100302 11625 webserver.cc:460] Webserver started at http://127.21.28.65:46489/ using document root <none> and password file <none>
I20260808 06:59:35.100560 11625 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:35.100636 11625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:35.102272 11625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:59:35.103060 11692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:35.103230 11625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:35.103317 11625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "138de8155d4142f3acf3c215f4de8b12"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "4973b456fc69cda36e9d45feff07b42d"
server_key_iv: "be64b624ad8f2f11def2287cb958fba2"
server_key_version: "encryptionkey@0"
I20260808 06:59:35.103643 11625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:35.120112 11625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:35.122150 11625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:35.122277 11625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:35.122484 11625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:35.122902 11702 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 06:59:35.124828 11625 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 06:59:35.124876 11625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 06:59:35.124907 11625 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 06:59:35.125516 11625 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 06:59:35.125548 11625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 06:59:35.125643 11702 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Bootstrap starting.
I20260808 06:59:35.132339 11625 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:33111
I20260808 06:59:35.132452 11847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:33111 every 8 connection(s)
I20260808 06:59:35.132719 11625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:59:35.136413 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 11625
I20260808 06:59:35.136673 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:40403
--local_ip_for_outbound_sockets=127.21.28.66
--tserver_master_addrs=127.21.28.126:36283
--webserver_port=35841
--webserver_interface=127.21.28.66
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:59:35.137712 11702 log.cc:824] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:35.140522 11702 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:59:35.140856 11702 tablet_bootstrap.cc:492] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Bootstrap complete.
I20260808 06:59:35.141939 11702 ts_tablet_manager.cc:1397] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260808 06:59:35.143291 11702 raft_consensus.cc:348] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:35.143605 11702 raft_consensus.cc:729] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138de8155d4142f3acf3c215f4de8b12, State: Initialized, Role: FOLLOWER
I20260808 06:59:35.143803 11702 consensus_queue.cc:260] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:35.144096 11702 ts_tablet_manager.cc:1428] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260808 06:59:35.191238 11850 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:59:35.133370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:38141)
0808 06:59:35.133569 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.133603 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:35.134031 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:35.191029 (+ 56998us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":119,"threads_started":1}
W20260808 06:59:35.191495 11848 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:36283 (0 consecutive failures): Network error: Failed to ping master at 127.21.28.126:36283: Client connection negotiation failed: client connection to 127.21.28.126:36283: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 06:59:35.191854 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:35.191753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:38525)
0808 06:59:35.191796 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.191833 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 06:59:35.192176 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:35.192100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:51443)
0808 06:59:35.192138 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.192160 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 06:59:35.192278 11848 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:59:35.219197 11852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:35.219372 11852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:35.219393 11852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:35.220557 11852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:35.220598 11852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:35.220623 11852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:59:35.221969 11852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:40403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=35841
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:35.222150 11852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:35.222347 11852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:59:35.226990 11852 server_base.cc:1034] running on GCE node
W20260808 06:59:35.226936 11864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:35.226941 11866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:35.226972 11863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:35.227314 11852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:35.227507 11852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:35.228658 11852 hybrid_clock.cc:648] HybridClock initialized: now 1786172375228637 us; error 37 us; skew 500 ppm
Aug 08 06:59:35 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172375, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:35.230888 11852 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:59:35.231293 11852 webserver.cc:460] Webserver started at http://127.21.28.66:35841/ using document root <none> and password file <none>
I20260808 06:59:35.231485 11852 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:35.231534 11852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:35.232832 11852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:35.233361 11875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:35.233512 11852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:35.233590 11852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "083b9226e90b4e72a1621b1a4603e586"
format_stamp: "Formatted at 2026-08-08 06:59:33 on dist-test-slave-nj21"
server_key: "fe5432b386f0c00cd3ff217c0e19d907"
server_key_iv: "5703cca9e971c2045a29997ee0b41a79"
server_key_version: "encryptionkey@0"
I20260808 06:59:35.233880 11852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:35.247073 11852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:35.248749 11852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:35.248875 11852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:35.249074 11852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:35.249334 11852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:35.249367 11852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:35.249394 11852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:35.249432 11852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:35.255000 11852 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40403
I20260808 06:59:35.255061 12038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40403 every 8 connection(s)
I20260808 06:59:35.255354 11852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 06:59:35.256313 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:35.255961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:56481)
0808 06:59:35.256105 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.256176 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
W20260808 06:59:35.256668 12039 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:36283 (0 consecutive failures): Network error: Failed to ping master at 127.21.28.126:36283: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
I20260808 06:59:35.256913 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:35.256827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:35847)
0808 06:59:35.256863 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.256896 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 06:59:35.257124 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:35.257064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:36991)
0808 06:59:35.257093 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.257112 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260808 06:59:35.257201 12039 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 06:59:35.262738 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 11852
I20260808 06:59:35.262974 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:35863
--local_ip_for_outbound_sockets=127.21.28.67
--tserver_master_addrs=127.21.28.126:36283
--webserver_port=44879
--webserver_interface=127.21.28.67
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:35.342178 12046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:35.342367 12046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:35.342396 12046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:35.343489 12046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:35.343528 12046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:35.343560 12046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:59:35.344993 12046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:35863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=44879
--tserver_master_addrs=127.21.28.126:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:35.345212 12046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:35.345412 12046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:35.350150 12056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:35.350172 12060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:35.350172 12058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:35.350314 12046 server_base.cc:1034] running on GCE node
I20260808 06:59:35.350502 12046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:35.350706 12046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:35.351830 12046 hybrid_clock.cc:648] HybridClock initialized: now 1786172375351815 us; error 30 us; skew 500 ppm
Aug 08 06:59:35 dist-test-slave-nj21 krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172375, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:35.354089 12046 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:59:35.354460 12046 webserver.cc:460] Webserver started at http://127.21.28.67:44879/ using document root <none> and password file <none>
I20260808 06:59:35.354640 12046 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:35.354681 12046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:35.355891 12046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:59:35.356396 12073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:35.356529 12046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:35.356602 12046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "526e76d4ea62488384543c3b398e060d"
format_stamp: "Formatted at 2026-08-08 06:59:34 on dist-test-slave-nj21"
server_key: "34ff11ef621b8f727a8df2b1cebf0dbf"
server_key_iv: "349e9647a61de739055e9e5b25d8e424"
server_key_version: "encryptionkey@0"
I20260808 06:59:35.356882 12046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:35.366247 12046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:35.367991 12046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:35.368105 12046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:35.368296 12046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:35.368561 12046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:35.368592 12046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:35.368618 12046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:35.368642 12046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:35.374200 12046 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:35863
I20260808 06:59:35.374253 12245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:35863 every 8 connection(s)
I20260808 06:59:35.374528 12046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:59:35.375439 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:35.375102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:53909)
0808 06:59:35.375232 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.375299 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
W20260808 06:59:35.375815 12246 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:36283 (0 consecutive failures): Network error: Failed to ping master at 127.21.28.126:36283: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
I20260808 06:59:35.376052 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:35.375975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:54247)
0808 06:59:35.376011 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.376036 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260808 06:59:35.376235 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:35.376189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:51441)
0808 06:59:35.376207 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:35.376224 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260808 06:59:35.376312 12246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 06:59:35.377674 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 12046
I20260808 06:59:36.192801 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:36.192592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:39203)
0808 06:59:36.192705 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:36.192736 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260808 06:59:36.257648 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:36.257464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:41847)
0808 06:59:36.257563 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:36.257595 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260808 06:59:36.356711 12255 raft_consensus.cc:482] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:36.356833 12255 raft_consensus.cc:504] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:36.357168 12255 leader_election.cc:288] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
Aug 08 06:59:36 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172375, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.66@KRBTEST.COM
Aug 08 06:59:36 dist-test-slave-nj21 krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172375, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.67@KRBTEST.COM
I20260808 06:59:36.366951 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:36.357434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:36713 (local address 127.21.28.67:35863)
0808 06:59:36.357557 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:36.357569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:36.357592 (+ 23us) server_negotiation.cc:407] Connection header received
0808 06:59:36.357997 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:36.357999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:36.358052 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:36.358200 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:36.358959 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:36.359930 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:36.360868 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:36.361271 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:36.361373 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:36.364086 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:36.364090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:36.364090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:36.364100 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:36.365545 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:36.365878 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:36.365880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:36.365881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:36.365976 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:36.366163 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:36.366164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:36.366164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:36.366385 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:36.366459 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:36.366732 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 06:59:36.366844 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260808 06:59:36.366966 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:36.357434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:55807 (local address 127.21.28.66:40403)
0808 06:59:36.357590 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:36.357599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:36.357628 (+ 29us) server_negotiation.cc:407] Connection header received
0808 06:59:36.358109 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:36.358111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:36.358155 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:36.358395 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:36.358987 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:36.359926 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:36.360862 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:36.361253 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:36.361339 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:36.363653 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:36.363656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:36.363656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:36.363665 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:36.365358 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:36.365808 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:36.365811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:36.365811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:36.365929 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:36.366167 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:36.366168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:36.366169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:36.366386 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:36.366460 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:36.366729 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 06:59:36.366846 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":53,"threads_started":1}
I20260808 06:59:36.367028 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:36.357310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40403 (local address 127.21.28.65:55807)
0808 06:59:36.357362 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:36.357372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:36.357440 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:36.358386 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:36.358387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:36.358670 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:36.358849 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:36.358856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:36.360013 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:36.360014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:36.360785 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:36.360787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:36.361335 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:36.361336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:36.361451 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:36.362023 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:36.362027 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:36.363561 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:36.365452 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:36.365459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:36.365461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:36.365733 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:36.365991 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:36.365992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:36.365992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:36.366107 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:36.366453 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:36.366459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:36.366580 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 06:59:36.366666 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 06:59:36.366914 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":134}
I20260808 06:59:36.366955 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:36.357329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:35863 (local address 127.21.28.65:36713)
0808 06:59:36.357485 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:36.357495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:36.357547 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:36.358246 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:36.358249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:36.358661 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:36.358848 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:36.358853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:36.360043 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:36.360044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:36.360787 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:36.360789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:36.361347 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:36.361348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:36.361449 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:36.361968 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:36.361975 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:36.363986 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:36.365617 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:36.365617 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:36.365618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:36.365806 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:36.366041 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:36.366041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:36.366042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:36.366107 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:36.366455 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:36.366461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:36.366580 (+ 119us) client_negotiation.cc:769] Sending connection context
0808 06:59:36.366667 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 06:59:36.366842 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":59,"thread_start_us":89,"threads_started":1}
W20260808 06:59:36.367868 11694 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
W20260808 06:59:36.368256 11695 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
I20260808 06:59:36.368299 11695 leader_election.cc:302] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138de8155d4142f3acf3c215f4de8b12; no voters: 083b9226e90b4e72a1621b1a4603e586, 526e76d4ea62488384543c3b398e060d
I20260808 06:59:36.368436 12255 raft_consensus.cc:2726] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:59:36.376722 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:36.376551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:53999)
0808 06:59:36.376658 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:36.376693 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260808 06:59:37.193472 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:37.193229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:51225)
0808 06:59:37.193372 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:37.193422 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260808 06:59:37.258153 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:37.257984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:59361)
0808 06:59:37.258088 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:37.258118 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260808 06:59:37.377213 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:37.377041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:59241)
0808 06:59:37.377147 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:37.377178 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260808 06:59:38.194089 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:38.193878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:59111)
0808 06:59:38.194008 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:38.194040 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260808 06:59:38.258678 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:38.258500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:40081)
0808 06:59:38.258606 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:38.258637 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260808 06:59:38.377766 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:38.377564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:35645)
0808 06:59:38.377695 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:38.377726 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260808 06:59:38.514982 12273 raft_consensus.cc:482] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:38.515079 12273 raft_consensus.cc:504] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:38.515239 12273 leader_election.cc:288] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
W20260808 06:59:38.515632 11695 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
W20260808 06:59:38.515784 11694 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
I20260808 06:59:38.515841 11694 leader_election.cc:302] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138de8155d4142f3acf3c215f4de8b12; no voters: 083b9226e90b4e72a1621b1a4603e586, 526e76d4ea62488384543c3b398e060d
I20260808 06:59:38.515906 12273 raft_consensus.cc:2726] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:59:39.194682 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:39.194481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:34407)
0808 06:59:39.194604 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:39.194636 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260808 06:59:39.259222 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:39.259045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:49665)
0808 06:59:39.259154 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:39.259185 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260808 06:59:39.378270 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:39.378096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:35485)
0808 06:59:39.378201 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:39.378233 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260808 06:59:40.195355 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:40.195116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:57177)
0808 06:59:40.195267 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:40.195307 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260808 06:59:40.259771 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:40.259564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:44759)
0808 06:59:40.259696 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:40.259731 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260808 06:59:40.378803 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:40.378623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:57883)
0808 06:59:40.378732 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:40.378763 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260808 06:59:40.820422 12282 raft_consensus.cc:482] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:40.820518 12282 raft_consensus.cc:504] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:40.820685 12282 leader_election.cc:288] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
W20260808 06:59:40.821036 11695 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
W20260808 06:59:40.821125 11694 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
I20260808 06:59:40.821168 11694 leader_election.cc:302] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138de8155d4142f3acf3c215f4de8b12; no voters: 083b9226e90b4e72a1621b1a4603e586, 526e76d4ea62488384543c3b398e060d
I20260808 06:59:40.821233 12282 raft_consensus.cc:2726] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:59:41.195977 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:41.195783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:53665)
0808 06:59:41.195894 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:41.195928 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260808 06:59:41.260327 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:41.260138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:54641)
0808 06:59:41.260254 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:41.260288 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260808 06:59:41.379357 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:41.379174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:51599)
0808 06:59:41.379283 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:41.379317 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260808 06:59:42.196640 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:42.196446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:51653)
0808 06:59:42.196560 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:42.196595 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260808 06:59:42.260941 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:42.260747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:60853)
0808 06:59:42.260868 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:42.260902 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260808 06:59:42.379931 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:42.379728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:41965)
0808 06:59:42.379857 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:42.379892 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260808 06:59:42.674038 12293 raft_consensus.cc:482] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:42.674152 12293 raft_consensus.cc:504] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:42.674350 12293 leader_election.cc:288] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
W20260808 06:59:42.674688 11695 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
W20260808 06:59:42.674777 11694 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
I20260808 06:59:42.674819 11694 leader_election.cc:302] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138de8155d4142f3acf3c215f4de8b12; no voters: 083b9226e90b4e72a1621b1a4603e586, 526e76d4ea62488384543c3b398e060d
I20260808 06:59:42.674878 12293 raft_consensus.cc:2726] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:59:43.197186 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:43.196987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:38803)
0808 06:59:43.197096 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:43.197130 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260808 06:59:43.261511 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:43.261312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:58877)
0808 06:59:43.261428 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:43.261466 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260808 06:59:43.380472 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:43.380282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:57093)
0808 06:59:43.380398 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:43.380433 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260808 06:59:44.197768 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:44.197572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:37715)
0808 06:59:44.197686 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:44.197719 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260808 06:59:44.262061 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:44.261873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:34969)
0808 06:59:44.261987 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:44.262020 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260808 06:59:44.381054 12300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:44.380857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:59709)
0808 06:59:44.380980 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:44.381014 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260808 06:59:45.198395 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:45.198150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:32925)
0808 06:59:45.198301 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:45.198341 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260808 06:59:45.262619 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:45.262417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:49275)
0808 06:59:45.262541 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:45.262577 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260808 06:59:45.381650 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:45.381423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:42263)
0808 06:59:45.381575 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:45.381609 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260808 06:59:46.199018 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:46.198777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:55329)
0808 06:59:46.198925 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:46.198965 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260808 06:59:46.263217 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:46.263018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:38481)
0808 06:59:46.263142 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:46.263175 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260808 06:59:46.382246 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:46.382043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:48499)
0808 06:59:46.382171 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:46.382205 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260808 06:59:47.199620 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:47.199414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:41733)
0808 06:59:47.199538 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:47.199571 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260808 06:59:47.263823 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:47.263614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:35105)
0808 06:59:47.263745 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:47.263780 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260808 06:59:47.382802 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:47.382593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:59757)
0808 06:59:47.382721 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:47.382755 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260808 06:59:48.200201 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:48.199994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:53761)
0808 06:59:48.200120 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:48.200152 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260808 06:59:48.264511 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:48.264246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:44559)
0808 06:59:48.264428 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:48.264461 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260808 06:59:48.383374 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:48.383188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:35785)
0808 06:59:48.383300 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:48.383335 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260808 06:59:49.200855 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:49.200648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:54527)
0808 06:59:49.200769 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:49.200805 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260808 06:59:49.265177 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:49.264961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:50523)
0808 06:59:49.265093 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:49.265131 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260808 06:59:49.383958 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:49.383766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:54543)
0808 06:59:49.383881 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:49.383919 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260808 06:59:50.098361 12320 raft_consensus.cc:482] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:50.098465 12320 raft_consensus.cc:504] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b9226e90b4e72a1621b1a4603e586" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40403 } } peers { permanent_uuid: "526e76d4ea62488384543c3b398e060d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 35863 } } peers { permanent_uuid: "138de8155d4142f3acf3c215f4de8b12" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33111 } }
I20260808 06:59:50.098639 12320 leader_election.cc:288] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403), 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863)
W20260808 06:59:50.099033 11694 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 083b9226e90b4e72a1621b1a4603e586 (127.21.28.66:40403): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
W20260808 06:59:50.099124 11695 leader_election.cc:341] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 526e76d4ea62488384543c3b398e060d (127.21.28.67:35863): Not found: Tablet not found: c524e48b474847a89169473e2e38c112
I20260808 06:59:50.099165 11695 leader_election.cc:302] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138de8155d4142f3acf3c215f4de8b12; no voters: 083b9226e90b4e72a1621b1a4603e586, 526e76d4ea62488384543c3b398e060d
I20260808 06:59:50.099238 12320 raft_consensus.cc:2726] T c524e48b474847a89169473e2e38c112 P 138de8155d4142f3acf3c215f4de8b12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:59:50.201485 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:50.201296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.65:54609)
0808 06:59:50.201398 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:50.201439 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260808 06:59:50.265836 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:50.265629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.66:60007)
0808 06:59:50.265751 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:50.265787 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260808 06:59:50.382921 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 11625
I20260808 06:59:50.389292 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:50.388419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36283 (local address 127.21.28.67:47575)
0808 06:59:50.389201 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:50.389241 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":712,"threads_started":1}
I20260808 06:59:50.393055 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 11852
I20260808 06:59:50.401424 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 12046
2026-08-08T06:59:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18030 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:59:50 dist-test-slave-nj21 krb5kdc[12328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:59:50 dist-test-slave-nj21 krb5kdc[12328](info): set up 2 sockets
Aug 08 06:59:50 dist-test-slave-nj21 krb5kdc[12328](info): commencing operation
krb5kdc: starting...
W20260808 06:59:51.530943 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 06:59:51 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 06:59:51.638837 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46061
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:51.720176 12351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:51.720362 12351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:51.720391 12351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:51.721469 12351 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 06:59:51.721506 12351 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 06:59:51.721524 12351 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 06:59:51.721537 12351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:59:51.721552 12351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:51.721571 12351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:51.721593 12351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:59:51.721617 12351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:59:51.723344 12351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:51.723619 12351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:51.723858 12351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:51.728828 12363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:51.728883 12351 server_base.cc:1034] running on GCE node
W20260808 06:59:51.728835 12361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:51.728845 12360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:51.729238 12351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:51.729529 12351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:51.730665 12351 hybrid_clock.cc:648] HybridClock initialized: now 1786172391730650 us; error 32 us; skew 500 ppm
Aug 08 06:59:51 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172391, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:51.733078 12351 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 06:59:51.733477 12351 webserver.cc:460] Webserver started at http://127.21.28.126:42583/ using document root <none> and password file <none>
I20260808 06:59:51.733641 12351 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:51.733678 12351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:51.733764 12351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:51.734679 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "439d11b42cc4478fabfec020ce227ba9"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "408387d284995525093fa932c92fcab9"
server_key_iv: "881454d2091c9c59306dd7f168ad62c5"
server_key_version: "encryptionkey@0"
I20260808 06:59:51.734966 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "439d11b42cc4478fabfec020ce227ba9"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "408387d284995525093fa932c92fcab9"
server_key_iv: "881454d2091c9c59306dd7f168ad62c5"
server_key_version: "encryptionkey@0"
I20260808 06:59:51.736145 12351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 06:59:51.736758 12373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:51.736915 12351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:51.736999 12351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "439d11b42cc4478fabfec020ce227ba9"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "408387d284995525093fa932c92fcab9"
server_key_iv: "881454d2091c9c59306dd7f168ad62c5"
server_key_version: "encryptionkey@0"
I20260808 06:59:51.737051 12351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:51.750113 12351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:51.751632 12351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:51.751771 12351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:51.755330 12351 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46061
I20260808 06:59:51.755362 12453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46061 every 8 connection(s)
I20260808 06:59:51.755657 12351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 06:59:51.756476 12454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:51.758562 12454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Bootstrap starting.
I20260808 06:59:51.759107 12454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:51.759366 12454 log.cc:824] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:51.760010 12454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: No bootstrap required, opened a new log
I20260808 06:59:51.761518 12454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 06:59:51.761641 12454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:51.761672 12454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439d11b42cc4478fabfec020ce227ba9, State: Initialized, Role: FOLLOWER
I20260808 06:59:51.761795 12454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 06:59:51.761869 12454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:59:51.761900 12454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:59:51.761948 12454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:51.762431 12454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 06:59:51.762521 12454 leader_election.cc:302] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439d11b42cc4478fabfec020ce227ba9; no voters:
I20260808 06:59:51.762671 12454 leader_election.cc:288] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:59:51.762708 12460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:51.762852 12460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 1 LEADER]: Becoming Leader. State: Replica: 439d11b42cc4478fabfec020ce227ba9, State: Running, Role: LEADER
I20260808 06:59:51.762917 12454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:59:51.762964 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 12351
I20260808 06:59:51.763060 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 06:59:51.763026 12460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 06:59:51.763279 21617 external_mini_cluster.cc:1428] Setting key 6aa9adf8aeb37f0f23158318e305e093
I20260808 06:59:51.763382 12462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "439d11b42cc4478fabfec020ce227ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } } }
I20260808 06:59:51.763445 12462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:51.763586 12464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439d11b42cc4478fabfec020ce227ba9. Latest consensus state: current_term: 1 leader_uuid: "439d11b42cc4478fabfec020ce227ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } } }
I20260808 06:59:51.763638 12464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: This master's current role is: LEADER
I20260808 06:59:51.763969 12469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:59:51.764526 12469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:59:51.766080 12469 catalog_manager.cc:1347] Generated new cluster ID: 75ee1e815494439ca0d2d1ef39d79ba5
I20260808 06:59:51.766134 12469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 06:59:51 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:51.772919 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:51.763721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.21.28.126:46061)
0808 06:59:51.764056 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:51.764065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:51.764094 (+ 29us) server_negotiation.cc:407] Connection header received
0808 06:59:51.764203 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:51.764210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:51.764266 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:51.764408 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:51.765191 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:51.766367 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:51.767096 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:51.767497 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:51.767587 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:51.769676 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:51.769680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:51.769680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:51.769691 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:51.771548 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:51.771925 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:51.771928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:51.771928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:51.772005 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:51.772182 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:51.772183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:51.772183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:51.772388 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:51.772506 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:51.772628 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 06:59:51.772756 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":73,"threads_started":1}
I20260808 06:59:51.777552 12469 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:59:51.778151 12469 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:59:51.785874 12469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Generated new TSK 0
I20260808 06:59:51.786024 12469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 06:59:51.816536 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46061
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:51.913537 12497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:51.913722 12497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:51.913753 12497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:51.914824 12497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:51.914861 12497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:51.914896 12497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 06:59:51.916311 12497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:51.916540 12497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:51.916733 12497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:51.921248 12503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:51.921252 12509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:51.921263 12506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:51.921268 12497 server_base.cc:1034] running on GCE node
I20260808 06:59:51.921651 12497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:51.921867 12497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:51.923007 12497 hybrid_clock.cc:648] HybridClock initialized: now 1786172391922995 us; error 28 us; skew 500 ppm
Aug 08 06:59:51 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172391, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:51.925479 12497 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 06:59:51.925885 12497 webserver.cc:460] Webserver started at http://127.21.28.65:46499/ using document root <none> and password file <none>
I20260808 06:59:51.926087 12497 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:51.926132 12497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:51.926245 12497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:51.927172 12497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "f23559b7d9904310ac964cd824675129"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "2c54bfef8b4e1b2a505e9c12cf8ff8c9"
server_key_iv: "b6449189da075007d5a5d24ed6dbbbdd"
server_key_version: "encryptionkey@0"
I20260808 06:59:51.927490 12497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "f23559b7d9904310ac964cd824675129"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "2c54bfef8b4e1b2a505e9c12cf8ff8c9"
server_key_iv: "b6449189da075007d5a5d24ed6dbbbdd"
server_key_version: "encryptionkey@0"
I20260808 06:59:51.928683 12497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:59:51.929293 12520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:51.929437 12497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:51.929526 12497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "f23559b7d9904310ac964cd824675129"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "2c54bfef8b4e1b2a505e9c12cf8ff8c9"
server_key_iv: "b6449189da075007d5a5d24ed6dbbbdd"
server_key_version: "encryptionkey@0"
I20260808 06:59:51.929589 12497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:51.940217 12497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:51.941802 12497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:51.941924 12497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:51.942103 12497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:51.942353 12497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:51.942386 12497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:51.942421 12497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:51.942445 12497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:51.948001 12497 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:41275
I20260808 06:59:51.948057 12692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:41275 every 8 connection(s)
I20260808 06:59:51.948313 12497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 06:59:51.951085 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 12497
I20260808 06:59:51.951182 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 06:59:51.951359 21617 external_mini_cluster.cc:1428] Setting key 067e95c5a16431007a74b638e5a5d2e3
Aug 08 06:59:51 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172391, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:51.957280 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:51.948914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:42757 (local address 127.21.28.126:46061)
0808 06:59:51.948983 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:51.948986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:51.949144 (+ 158us) server_negotiation.cc:407] Connection header received
0808 06:59:51.949591 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:51.949592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:51.949604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:51.949647 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:51.950478 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:51.951029 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:51.952309 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:51.952566 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:51.952639 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:51.954681 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:51.954682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:51.954682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:51.954688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:51.956069 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:51.956441 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:51.956442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:51.956443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:51.956502 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:51.956707 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:51.956708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:51.956709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:51.956832 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:51.956913 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:51.957105 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 06:59:51.957188 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:59:51.957329 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:51.948886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:42757)
0808 06:59:51.949038 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:51.949071 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:51.949518 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:51.949702 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:51.949705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:51.950231 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:51.950403 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:51.950406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:51.951099 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:51.951099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:51.952225 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:51.952227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:51.952639 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:51.952639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:51.952741 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:51.953183 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:51.953191 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:51.954600 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:51.956146 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:51.956155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:51.956157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:51.956375 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:51.956562 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:51.956563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:51.956563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:51.956658 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:51.956890 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:51.956893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:51.956995 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 06:59:51.957068 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 06:59:51.957177 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260808 06:59:51.957782 12693 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46061
I20260808 06:59:51.957866 12693 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:51.958034 12693 heartbeater.cc:511] Master 127.21.28.126:46061 requested a full tablet report, sending...
I20260808 06:59:51.958495 12397 ts_manager.cc:194] Registered new tserver with Master: f23559b7d9904310ac964cd824675129 (127.21.28.65:41275)
I20260808 06:59:51.959056 12397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:42757
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 06:59:51.973819 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46061
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:52.052217 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:52.052408 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:52.052438 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:52.053514 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:52.053555 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:52.053588 12702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 06:59:52.054971 12702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:52.055198 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:52.055389 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:59:52.060187 12713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:52.060194 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:52.060287 12702 server_base.cc:1034] running on GCE node
W20260808 06:59:52.060194 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:52.060571 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:52.060757 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:52.061892 12702 hybrid_clock.cc:648] HybridClock initialized: now 1786172392061878 us; error 36 us; skew 500 ppm
Aug 08 06:59:52 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:52.064364 12702 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 06:59:52.064774 12702 webserver.cc:460] Webserver started at http://127.21.28.66:45357/ using document root <none> and password file <none>
I20260808 06:59:52.065073 12702 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:52.065136 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:52.065264 12702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:52.066574 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "a1927b504e0d4f1496430f95b5e6d3c4"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "7ef033d4bcbbbfeec03ae4ff8f472e2a"
server_key_iv: "aa293a659feb65fdc9c1b3061b379806"
server_key_version: "encryptionkey@0"
I20260808 06:59:52.066962 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "a1927b504e0d4f1496430f95b5e6d3c4"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "7ef033d4bcbbbfeec03ae4ff8f472e2a"
server_key_iv: "aa293a659feb65fdc9c1b3061b379806"
server_key_version: "encryptionkey@0"
I20260808 06:59:52.068456 12702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:59:52.069144 12722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:52.069307 12702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:52.069389 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "a1927b504e0d4f1496430f95b5e6d3c4"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "7ef033d4bcbbbfeec03ae4ff8f472e2a"
server_key_iv: "aa293a659feb65fdc9c1b3061b379806"
server_key_version: "encryptionkey@0"
I20260808 06:59:52.069464 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:52.082306 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:52.083973 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:52.084101 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:52.084304 12702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:52.084580 12702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:52.084614 12702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:52.084640 12702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:52.084657 12702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:52.090422 12702 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40827
I20260808 06:59:52.090461 12898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40827 every 8 connection(s)
I20260808 06:59:52.090728 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:59:52 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:52.097676 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 12702
I20260808 06:59:52.097769 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 06:59:52.097993 21617 external_mini_cluster.cc:1428] Setting key 54da19fe969195c4ea10ced5a56d0400
I20260808 06:59:52.099166 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.091495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:37619 (local address 127.21.28.126:46061)
0808 06:59:52.091559 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.091560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.091758 (+ 198us) server_negotiation.cc:407] Connection header received
0808 06:59:52.092267 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.092268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.092279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.092332 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:52.093171 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.093655 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.094539 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.094711 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.094751 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.096589 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:52.096590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:52.096590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:52.096596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:52.097851 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:52.098295 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:52.098297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:52.098297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:52.098358 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:52.098569 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:52.098569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:52.098570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:52.098668 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:52.098724 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.098980 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.099077 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:59:52.099299 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.091475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:37619)
0808 06:59:52.091625 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:52.091665 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:52.092172 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:52.092381 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:52.092383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:52.092924 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:52.093108 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.093112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.093713 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.093714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.094480 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.094482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.094762 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.094762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.094852 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.095238 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:52.095245 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:52.096505 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:52.097940 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:52.097947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:52.097949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:52.098224 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:52.098419 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:52.098419 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:52.098420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:52.098514 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:52.098732 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:52.098740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:52.098864 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 06:59:52.098953 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 06:59:52.099098 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260808 06:59:52.099926 12899 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46061
I20260808 06:59:52.100003 12899 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:52.100157 12899 heartbeater.cc:511] Master 127.21.28.126:46061 requested a full tablet report, sending...
I20260808 06:59:52.100515 12397 ts_manager.cc:194] Registered new tserver with Master: a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827)
I20260808 06:59:52.100865 12397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:37619
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 06:59:52.120565 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46061
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:59:52.199764 12908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:59:52.199949 12908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:59:52.199970 12908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:59:52.201061 12908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:59:52.201093 12908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:59:52.201118 12908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 06:59:52.202469 12908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 06:59:52.202634 12908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:59:52.202801 12908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:59:52.207433 12908 server_base.cc:1034] running on GCE node
W20260808 06:59:52.207386 12917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:52.207393 12919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:59:52.207386 12924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:59:52.207762 12908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:59:52.207970 12908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:59:52.209096 12908 hybrid_clock.cc:648] HybridClock initialized: now 1786172392209085 us; error 28 us; skew 500 ppm
Aug 08 06:59:52 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:59:52.211407 12908 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 06:59:52.211843 12908 webserver.cc:460] Webserver started at http://127.21.28.67:44663/ using document root <none> and password file <none>
I20260808 06:59:52.212039 12908 fs_manager.cc:359] Metadata directory not provided
I20260808 06:59:52.212081 12908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:59:52.212196 12908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:59:52.213160 12908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "16bb79f2c5fd43ec823f25fdd0814077"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "5efa1dd023337ad501e27d04b34a3f03"
server_key_iv: "503434ad3d092e8c8710096c2071bedb"
server_key_version: "encryptionkey@0"
I20260808 06:59:52.213471 12908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "16bb79f2c5fd43ec823f25fdd0814077"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "5efa1dd023337ad501e27d04b34a3f03"
server_key_iv: "503434ad3d092e8c8710096c2071bedb"
server_key_version: "encryptionkey@0"
I20260808 06:59:52.214690 12908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 06:59:52.215374 12935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:52.215543 12908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 06:59:52.215623 12908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "16bb79f2c5fd43ec823f25fdd0814077"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "5efa1dd023337ad501e27d04b34a3f03"
server_key_iv: "503434ad3d092e8c8710096c2071bedb"
server_key_version: "encryptionkey@0"
I20260808 06:59:52.215687 12908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:59:52.234824 12908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:59:52.236773 12908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:59:52.236881 12908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:59:52.237066 12908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:59:52.237303 12908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:59:52.237331 12908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:52.237351 12908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:59:52.237370 12908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:59:52.243029 12908 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:38411
I20260808 06:59:52.243090 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:38411 every 8 connection(s)
I20260808 06:59:52.243335 12908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 06:59:52.244825 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 12908
I20260808 06:59:52.244930 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 06:59:52.245108 21617 external_mini_cluster.cc:1428] Setting key 74d037fa091950ff2bc8572e99601529
Aug 08 06:59:52 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 06:59:52.251564 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.243948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:38171 (local address 127.21.28.126:46061)
0808 06:59:52.244025 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.244027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.244173 (+ 146us) server_negotiation.cc:407] Connection header received
0808 06:59:52.244656 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.244657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.244690 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.244740 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:52.245774 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.246275 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.247214 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.247414 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.247469 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.249327 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:52.249328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:52.249328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:52.249335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:52.250362 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:52.250713 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:52.250714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:52.250715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:52.250770 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:52.250981 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:52.250981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:52.250982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:52.251094 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:52.251151 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.251396 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.251477 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:59:52.251749 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.243924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:38171)
0808 06:59:52.244061 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:52.244093 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:52.244563 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:52.244806 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:52.244809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:52.245455 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:59:52.245703 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.245708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.246354 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.246356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.247129 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.247131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.247482 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.247483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.247611 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.247980 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:59:52.247987 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:59:52.249247 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:59:52.250440 (+ 1193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:52.250446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:52.250448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:52.250649 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:52.250825 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:59:52.250826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:59:52.250826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:59:52.250926 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:59:52.251148 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:59:52.251154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:59:52.251280 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 06:59:52.251363 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:59:52.251520 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260808 06:59:52.252132 13097 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46061
I20260808 06:59:52.252211 13097 heartbeater.cc:464] Registering TS with master...
I20260808 06:59:52.252344 13097 heartbeater.cc:511] Master 127.21.28.126:46061 requested a full tablet report, sending...
I20260808 06:59:52.252687 12397 ts_manager.cc:194] Registered new tserver with Master: 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411)
I20260808 06:59:52.253046 12397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:38171
I20260808 06:59:52.256621 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:59:52.262821 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.257887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.21.28.126:46061)
0808 06:59:52.257930 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.257932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.258007 (+ 75us) server_negotiation.cc:407] Connection header received
0808 06:59:52.258046 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.258046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.258052 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.258096 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:59:52.258576 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.259012 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.259655 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.259865 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.259912 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.260912 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:59:52.260921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:59:52.260921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:59:52.260927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:59:52.261847 (+ 920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:52.262132 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:52.262133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:52.262134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:52.262182 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:59:52.262338 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:59:52.262338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:59:52.262339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:59:52.262470 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:59:52.262556 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.262673 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.262742 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 06:59:52.264331 12397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:59:52.265861 12397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:59:52.271607 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.267721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41275 (local address 127.0.0.1:55074)
0808 06:59:52.267862 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:52.267888 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:52.267956 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:52.268351 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:52.268354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:52.268357 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:52.268414 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.268416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.269418 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.269419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.270638 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.270641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.271321 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.271322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.271410 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.271424 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 06:59:52.271486 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 06:59:52.271538 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":27,"thread_start_us":51,"threads_started":1}
I20260808 06:59:52.271621 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.267828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.21.28.65:41275)
0808 06:59:52.268206 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.268209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.268217 (+ 8us) server_negotiation.cc:407] Connection header received
0808 06:59:52.268247 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.268249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.268259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.268332 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:52.268520 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.269325 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.270726 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.271246 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.271308 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.271364 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.271508 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.271548 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":85,"threads_started":1}
I20260808 06:59:52.271834 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.267731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:38411 (local address 127.0.0.1:54846)
0808 06:59:52.267999 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:52.268010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:52.268058 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:52.268377 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:52.268378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:52.268381 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:52.268436 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.268438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.269648 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.269649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.270704 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.270706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.271605 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.271606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.271694 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.271698 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:59:52.271727 (+ 29us) client_negotiation.cc:240] Negotiation successful
0808 06:59:52.271765 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":52,"threads_started":1}
I20260808 06:59:52.272014 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.267924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40827 (local address 127.0.0.1:58734)
0808 06:59:52.268436 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:52.268446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:52.268524 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:52.268780 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:52.268781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:52.268784 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:52.268819 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.268821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.269937 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.269938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.270777 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.270779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.271705 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.271706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.271765 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.271768 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:59:52.271810 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 06:59:52.271841 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":46,"threads_started":1}
I20260808 06:59:52.272037 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.268034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.21.28.66:40827)
0808 06:59:52.268193 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.268196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.268497 (+ 301us) server_negotiation.cc:407] Connection header received
0808 06:59:52.268609 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.268612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.268627 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.268677 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:52.268890 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.269845 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.270849 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.271536 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.271593 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.271645 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.271923 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.271966 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260808 06:59:52.272367 12590 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5dff56143b4494a16e5df6c0f3be6e (DEFAULT_TABLE table=test-table [id=e067b80e9e674ddf9abebf74c39f9fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:52.272678 12590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5dff56143b4494a16e5df6c0f3be6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:52.272779 12797 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5dff56143b4494a16e5df6c0f3be6e (DEFAULT_TABLE table=test-table [id=e067b80e9e674ddf9abebf74c39f9fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:52.273105 12797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5dff56143b4494a16e5df6c0f3be6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:52.275177 13125 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Bootstrap starting.
I20260808 06:59:52.275454 13128 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Bootstrap starting.
I20260808 06:59:52.275867 13125 tablet_bootstrap.cc:654] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:52.275915 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.268027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.21.28.67:38411)
0808 06:59:52.268247 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.268250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.268257 (+ 7us) server_negotiation.cc:407] Connection header received
0808 06:59:52.268284 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.268285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.268296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.268364 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:52.268512 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.269546 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.270761 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.271504 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.272331 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.272393 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.272425 (+ 32us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.275845 (+ 3420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260808 06:59:52.276139 13128 tablet_bootstrap.cc:654] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:52.276453 13125 log.cc:824] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:52.276600 13128 log.cc:824] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:52.276538 13008 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5dff56143b4494a16e5df6c0f3be6e (DEFAULT_TABLE table=test-table [id=e067b80e9e674ddf9abebf74c39f9fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:59:52.276844 13008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5dff56143b4494a16e5df6c0f3be6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:59:52.277309 13125 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: No bootstrap required, opened a new log
I20260808 06:59:52.277364 13128 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: No bootstrap required, opened a new log
I20260808 06:59:52.277371 13125 ts_tablet_manager.cc:1397] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:52.277436 13128 ts_tablet_manager.cc:1397] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 06:59:52.278715 13125 raft_consensus.cc:348] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.278851 13125 raft_consensus.cc:374] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:52.278885 13125 raft_consensus.cc:729] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23559b7d9904310ac964cd824675129, State: Initialized, Role: FOLLOWER
I20260808 06:59:52.279008 13125 consensus_queue.cc:260] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.279290 13125 ts_tablet_manager.cc:1428] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:52.279425 12693 heartbeater.cc:503] Master 127.21.28.126:46061 was elected leader, sending a full tablet report...
I20260808 06:59:52.279337 13128 raft_consensus.cc:348] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.279497 13128 raft_consensus.cc:374] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:52.279536 13128 raft_consensus.cc:729] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1927b504e0d4f1496430f95b5e6d3c4, State: Initialized, Role: FOLLOWER
I20260808 06:59:52.279696 13128 consensus_queue.cc:260] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.279966 12899 heartbeater.cc:503] Master 127.21.28.126:46061 was elected leader, sending a full tablet report...
I20260808 06:59:52.280184 13133 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: Bootstrap starting.
I20260808 06:59:52.280215 13128 ts_tablet_manager.cc:1428] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 06:59:52.280807 13133 tablet_bootstrap.cc:654] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: Neither blocks nor log segments found. Creating new log.
I20260808 06:59:52.281242 13133 log.cc:824] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: Log is configured to *not* fsync() on all Append() calls
I20260808 06:59:52.281821 13133 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: No bootstrap required, opened a new log
I20260808 06:59:52.281870 13133 ts_tablet_manager.cc:1397] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:59:52.283110 13133 raft_consensus.cc:348] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.283233 13133 raft_consensus.cc:374] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:59:52.283274 13133 raft_consensus.cc:729] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bb79f2c5fd43ec823f25fdd0814077, State: Initialized, Role: FOLLOWER
I20260808 06:59:52.283407 13133 consensus_queue.cc:260] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.283613 13133 ts_tablet_manager.cc:1428] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:59:52.283648 13097 heartbeater.cc:503] Master 127.21.28.126:46061 was elected leader, sending a full tablet report...
W20260808 06:59:52.341372 12900 tablet.cc:2369] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:59:52.395153 13134 raft_consensus.cc:482] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:59:52.395272 13134 raft_consensus.cc:504] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.395694 13134 leader_election.cc:288] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23559b7d9904310ac964cd824675129 (127.21.28.65:41275), 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411)
I20260808 06:59:52.398272 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.395914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:60423 (local address 127.21.28.65:41275)
0808 06:59:52.395959 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.395961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.395973 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:59:52.396000 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.396001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.396010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.396068 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:52.396224 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.396692 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.397596 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.397961 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.398023 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.398032 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.398172 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.398207 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:59:52.398272 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.395833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41275 (local address 127.21.28.66:60423)
0808 06:59:52.395888 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:52.395899 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:52.395940 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:52.396085 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:52.396086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:52.396088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:52.396160 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.396161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.396759 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.396760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.397506 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.397508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.398029 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.398029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.398108 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.398111 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:59:52.398156 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 06:59:52.398198 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 06:59:52.398341 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.395939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:51699 (local address 127.21.28.67:38411)
0808 06:59:52.395981 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.395983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.396092 (+ 109us) server_negotiation.cc:407] Connection header received
0808 06:59:52.396123 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.396124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.396133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.396179 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:59:52.396324 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.396779 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.397664 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.398038 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.398096 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.398104 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.398238 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.398273 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 06:59:52.398432 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.395833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:38411 (local address 127.21.28.66:51699)
0808 06:59:52.396028 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:52.396035 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 06:59:52.396077 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:59:52.396188 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:59:52.396189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:59:52.396190 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:59:52.396266 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.396268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.396851 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.396852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.397586 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:59:52.397587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.398110 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.398111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:59:52.398178 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.398181 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:59:52.398224 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 06:59:52.398263 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":11,"thread_start_us":101,"threads_started":1}
I20260808 06:59:52.398676 12626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5dff56143b4494a16e5df6c0f3be6e" candidate_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23559b7d9904310ac964cd824675129" is_pre_election: true
I20260808 06:59:52.398726 13033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5dff56143b4494a16e5df6c0f3be6e" candidate_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bb79f2c5fd43ec823f25fdd0814077" is_pre_election: true
I20260808 06:59:52.398831 12626 raft_consensus.cc:2454] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1927b504e0d4f1496430f95b5e6d3c4 in term 0.
I20260808 06:59:52.398839 13033 raft_consensus.cc:2454] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1927b504e0d4f1496430f95b5e6d3c4 in term 0.
I20260808 06:59:52.399044 12724 leader_election.cc:302] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bb79f2c5fd43ec823f25fdd0814077, a1927b504e0d4f1496430f95b5e6d3c4; no voters:
I20260808 06:59:52.399211 13134 raft_consensus.cc:2781] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:59:52.399261 13134 raft_consensus.cc:482] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:59:52.399276 13134 raft_consensus.cc:3037] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:52.400045 13134 raft_consensus.cc:504] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.400162 13134 leader_election.cc:288] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [CANDIDATE]: Term 1 election: Requested vote from peers f23559b7d9904310ac964cd824675129 (127.21.28.65:41275), 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411)
I20260808 06:59:52.400292 12626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5dff56143b4494a16e5df6c0f3be6e" candidate_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23559b7d9904310ac964cd824675129"
I20260808 06:59:52.400317 13033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5dff56143b4494a16e5df6c0f3be6e" candidate_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bb79f2c5fd43ec823f25fdd0814077"
I20260808 06:59:52.400368 12626 raft_consensus.cc:3037] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:52.400377 13033 raft_consensus.cc:3037] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:59:52.401055 13033 raft_consensus.cc:2454] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1927b504e0d4f1496430f95b5e6d3c4 in term 1.
I20260808 06:59:52.401067 12626 raft_consensus.cc:2454] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1927b504e0d4f1496430f95b5e6d3c4 in term 1.
I20260808 06:59:52.401211 12724 leader_election.cc:302] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bb79f2c5fd43ec823f25fdd0814077, a1927b504e0d4f1496430f95b5e6d3c4; no voters:
I20260808 06:59:52.401298 13134 raft_consensus.cc:2781] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:59:52.401440 13134 raft_consensus.cc:686] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 LEADER]: Becoming Leader. State: Replica: a1927b504e0d4f1496430f95b5e6d3c4, State: Running, Role: LEADER
I20260808 06:59:52.401523 13134 consensus_queue.cc:237] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 06:59:52.402293 12396 catalog_manager.cc:5495] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66). New cstate: current_term: 1 leader_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } health_report { overall_health: HEALTHY } } }
I20260808 06:59:52.413203 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.411311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.21.28.66:40827)
0808 06:59:52.411356 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 06:59:52.411357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 06:59:52.411387 (+ 30us) server_negotiation.cc:407] Connection header received
0808 06:59:52.411418 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:59:52.411419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:59:52.411433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:59:52.411478 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:59:52.411579 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.412020 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.412676 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:59:52.412845 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:59:52.412886 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:59:52.412982 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:59:52.413063 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:59:52.413113 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 06:59:52.413131 (+ 18us) server_negotiation.cc:299] Negotiation successful
0808 06:59:52.413144 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 06:59:52.414644 12626 raft_consensus.cc:1261] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Refusing update from remote peer a1927b504e0d4f1496430f95b5e6d3c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:52.414652 13033 raft_consensus.cc:1261] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 FOLLOWER]: Refusing update from remote peer a1927b504e0d4f1496430f95b5e6d3c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:59:52.414971 13141 consensus_queue.cc:1035] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:52.415091 13134 consensus_queue.cc:1035] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:59:52.417059 13145 mvcc.cc:205] Tried to move back new op lower bound from 7316162119328378880 to 7316162119277117440. Current Snapshot: MvccSnapshot[applied={T|T < 7316162119328378880}]
I20260808 06:59:52.419234 13148 mvcc.cc:205] Tried to move back new op lower bound from 7316162119328378880 to 7316162119277117440. Current Snapshot: MvccSnapshot[applied={T|T < 7316162119328378880}]
I20260808 06:59:52.419576 12396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:59:52.419661 12396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:59:52.420392 12396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Sending DeleteTablet for 3 replicas of tablet 5d5dff56143b4494a16e5df6c0f3be6e
I20260808 06:59:52.420727 12797 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5dff56143b4494a16e5df6c0f3be6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:52 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:58734
I20260808 06:59:52.420724 12590 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5dff56143b4494a16e5df6c0f3be6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:52 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:55074
I20260808 06:59:52.420949 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 12497
I20260808 06:59:52.420995 13008 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5dff56143b4494a16e5df6c0f3be6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:52 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:54846
I20260808 06:59:52.421208 13159 tablet_replica.cc:330] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: stopping tablet replica
I20260808 06:59:52.421300 13159 raft_consensus.cc:2229] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:59:52.423305 13158 tablet_replica.cc:330] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: stopping tablet replica
I20260808 06:59:52.423378 13158 raft_consensus.cc:2229] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:59:52.423448 13158 raft_consensus.cc:2258] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:52.431250 13160 tablet_replica.cc:330] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: stopping tablet replica
I20260808 06:59:52.431360 13160 raft_consensus.cc:2229] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:59:52.431464 13160 raft_consensus.cc:2258] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:59:52.431876 13158 ts_tablet_manager.cc:1905] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:52.431990 13160 ts_tablet_manager.cc:1905] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:59:52.433534 13160 ts_tablet_manager.cc:1918] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:52.433552 13158 ts_tablet_manager.cc:1918] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:59:52.433594 13160 log.cc:1196] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/5d5dff56143b4494a16e5df6c0f3be6e
I20260808 06:59:52.433612 13158 log.cc:1196] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/5d5dff56143b4494a16e5df6c0f3be6e
I20260808 06:59:52.433863 13160 ts_tablet_manager.cc:1939] T 5d5dff56143b4494a16e5df6c0f3be6e P 16bb79f2c5fd43ec823f25fdd0814077: Deleting consensus metadata
I20260808 06:59:52.433878 13158 ts_tablet_manager.cc:1939] T 5d5dff56143b4494a16e5df6c0f3be6e P a1927b504e0d4f1496430f95b5e6d3c4: Deleting consensus metadata
I20260808 06:59:52.434252 12377 catalog_manager.cc:4841] TS a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827): tablet 5d5dff56143b4494a16e5df6c0f3be6e (table test-table [id=e067b80e9e674ddf9abebf74c39f9fd7]) successfully deleted
I20260808 06:59:52.434340 12377 catalog_manager.cc:4841] TS 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411): tablet 5d5dff56143b4494a16e5df6c0f3be6e (table test-table [id=e067b80e9e674ddf9abebf74c39f9fd7]) successfully deleted
W20260808 06:59:52.435601 12375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:41275 (error 108)
I20260808 06:59:52.436163 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 12702
I20260808 06:59:52.436317 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:59:52.436191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41275 (local address 127.0.0.1:55090)
0808 06:59:52.436247 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 06:59:52.436286 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 06:59:52.444831 12375 catalog_manager.cc:4584] TS f23559b7d9904310ac964cd824675129 (127.21.28.65:41275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d5dff56143b4494a16e5df6c0f3be6e: Network error: Client connection negotiation failed: client connection to 127.21.28.65:41275: connect: Connection refused (error 111)
I20260808 06:59:52.454447 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 12908
I20260808 06:59:52.468101 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 12351
I20260808 07:00:12.479172 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46061
--webserver_interface=127.21.28.126
--webserver_port=42583
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:12.561617 13164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:12.561805 13164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:12.561825 13164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:12.562959 13164 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 07:00:12.562996 13164 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 07:00:12.563009 13164 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 07:00:12.563019 13164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:12.563030 13164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:12.563041 13164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:12.563054 13164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:12.563079 13164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:12.564656 13164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:12.564863 13164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:12.565085 13164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 07:00:12.573751 13164 server_base.cc:1034] running on GCE node
W20260808 07:00:12.573856 13172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:12.573887 13173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:12.573899 13175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:12.574226 13164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:12.574630 13164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:12.575733 13164 hybrid_clock.cc:648] HybridClock initialized: now 1786172412575715 us; error 33 us; skew 500 ppm
Aug 08 07:00:12 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:12.578420 13164 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:00:12.578824 13164 webserver.cc:460] Webserver started at http://127.21.28.126:42583/ using document root <none> and password file <none>
I20260808 07:00:12.579028 13164 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:12.579083 13164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:12.580438 13164 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260808 07:00:12.581071 13183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:12.581226 13164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:12.581321 13164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "439d11b42cc4478fabfec020ce227ba9"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "408387d284995525093fa932c92fcab9"
server_key_iv: "881454d2091c9c59306dd7f168ad62c5"
server_key_version: "encryptionkey@0"
I20260808 07:00:12.581739 13164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:12.603062 13164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:12.605175 13164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:12.605304 13164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:12.609293 13164 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46061
I20260808 07:00:12.609349 13261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46061 every 8 connection(s)
I20260808 07:00:12.609700 13164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:12.611135 13262 sys_catalog.cc:263] Verifying existing consensus state
I20260808 07:00:12.611792 13262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Bootstrap starting.
I20260808 07:00:12.615105 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 13164
I20260808 07:00:12.615387 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:41275
--local_ip_for_outbound_sockets=127.21.28.65
--tserver_master_addrs=127.21.28.126:46061
--webserver_port=46499
--webserver_interface=127.21.28.65
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 07:00:12.620167 13262 log.cc:824] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:12.622051 13262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 07:00:12.622331 13262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Bootstrap complete.
I20260808 07:00:12.624482 13262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 07:00:12.624697 13262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439d11b42cc4478fabfec020ce227ba9, State: Initialized, Role: FOLLOWER
I20260808 07:00:12.624840 13262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 07:00:12.624898 13262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:12.624943 13262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:12.624989 13262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 07:00:12.625957 13262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 07:00:12.626044 13262 leader_election.cc:302] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439d11b42cc4478fabfec020ce227ba9; no voters:
I20260808 07:00:12.626184 13262 leader_election.cc:288] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 07:00:12.626235 13269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 07:00:12.626360 13269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 2 LEADER]: Becoming Leader. State: Replica: 439d11b42cc4478fabfec020ce227ba9, State: Running, Role: LEADER
I20260808 07:00:12.626456 13262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:12.626444 13269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 07:00:12.626827 13271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439d11b42cc4478fabfec020ce227ba9. Latest consensus state: current_term: 2 leader_uuid: "439d11b42cc4478fabfec020ce227ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } } }
I20260808 07:00:12.626921 13271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:12.626842 13272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "439d11b42cc4478fabfec020ce227ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } } }
I20260808 07:00:12.627067 13272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:12.627103 13281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:12.627897 13281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:12.627996 13281 catalog_manager.cc:1259] Loaded cluster ID: 75ee1e815494439ca0d2d1ef39d79ba5
I20260808 07:00:12.628031 13281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:00:12.629601 13281 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:12.629799 13281 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Loaded TSK: 0
I20260808 07:00:12.652899 13281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Generated new TSK 1
I20260808 07:00:12.653019 13281 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 07:00:12.704859 13264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:12.705060 13264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:12.705082 13264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:12.706225 13264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:12.706267 13264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:12.706293 13264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 07:00:12.707758 13264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:41275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=46499
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:12.707978 13264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:12.708190 13264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:12.713966 13303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:12.713963 13302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:12.713963 13307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:12.714151 13264 server_base.cc:1034] running on GCE node
I20260808 07:00:12.714397 13264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:12.714618 13264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:12.715741 13264 hybrid_clock.cc:648] HybridClock initialized: now 1786172412715734 us; error 40 us; skew 500 ppm
Aug 08 07:00:12 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:12.718108 13264 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 07:00:12.718516 13264 webserver.cc:460] Webserver started at http://127.21.28.65:46499/ using document root <none> and password file <none>
I20260808 07:00:12.718719 13264 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:12.718783 13264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:12.720115 13264 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:12.720829 13316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:12.720986 13264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:12.721087 13264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "f23559b7d9904310ac964cd824675129"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "2c54bfef8b4e1b2a505e9c12cf8ff8c9"
server_key_iv: "b6449189da075007d5a5d24ed6dbbbdd"
server_key_version: "encryptionkey@0"
I20260808 07:00:12.721506 13264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:12.741736 13264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:12.743894 13264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:12.744024 13264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:12.744240 13264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:12.744665 13324 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 07:00:12.746574 13264 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 07:00:12.746675 13264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 07:00:12.746721 13264 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 07:00:12.747604 13264 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 07:00:12.747639 13264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 07:00:12.747747 13324 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Bootstrap starting.
I20260808 07:00:12.753854 13264 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:41275
I20260808 07:00:12.753958 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:41275 every 8 connection(s)
I20260808 07:00:12.754230 13264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 07:00:12.758901 13324 log.cc:824] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:12.760994 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 13264
I20260808 07:00:12.761222 13324 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 07:00:12.761265 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:40827
--local_ip_for_outbound_sockets=127.21.28.66
--tserver_master_addrs=127.21.28.126:46061
--webserver_port=45357
--webserver_interface=127.21.28.66
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 07:00:12.761516 13324 tablet_bootstrap.cc:492] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Bootstrap complete.
I20260808 07:00:12.762590 13324 ts_tablet_manager.cc:1397] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260808 07:00:12.763593 13324 raft_consensus.cc:348] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 07:00:12.763944 13324 raft_consensus.cc:729] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23559b7d9904310ac964cd824675129, State: Initialized, Role: FOLLOWER
I20260808 07:00:12.764102 13324 consensus_queue.cc:260] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } } peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } }
I20260808 07:00:12.764375 13324 ts_tablet_manager.cc:1428] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Aug 08 07:00:12 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:12.769456 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:12.754900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:40685)
0808 07:00:12.759372 (+ 4472us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:12.759419 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:12.759925 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:12.760375 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:12.760378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:12.760640 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:12.760917 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:12.760923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.762520 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.762521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:12.763452 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:12.763455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.764352 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.764353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:12.764498 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:12.765003 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:12.765011 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:12.766302 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:12.767959 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:12.767966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:12.767968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:12.768179 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:12.768402 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:12.768403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:12.768404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:12.768510 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:12.768843 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:12.768849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:12.768979 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 07:00:12.769064 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 07:00:12.769231 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4426,"thread_start_us":99,"threads_started":1}
I20260808 07:00:12.769459 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:12.754973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:40685 (local address 127.21.28.126:46061)
0808 07:00:12.759800 (+ 4827us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:12.759810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:12.759855 (+ 45us) server_negotiation.cc:407] Connection header received
0808 07:00:12.760125 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:12.760132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:12.760190 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:12.760350 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:12.761043 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.762404 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.763541 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.764233 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.764733 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:12.766397 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:12.766401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:12.766401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:12.766411 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:12.767888 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:12.768253 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:12.768256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:12.768256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:12.768343 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:12.768565 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:12.768566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:12.768567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:12.768781 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:12.768851 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:12.769122 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 07:00:12.769240 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4772,"thread_start_us":75,"threads_started":1}
I20260808 07:00:12.770183 13493 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46061
I20260808 07:00:12.770267 13493 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:12.770422 13493 heartbeater.cc:511] Master 127.21.28.126:46061 requested a full tablet report, sending...
I20260808 07:00:12.771063 13210 ts_manager.cc:194] Registered new tserver with Master: f23559b7d9904310ac964cd824675129 (127.21.28.65:41275)
I20260808 07:00:12.772024 13210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:40685
I20260808 07:00:12.772758 13493 heartbeater.cc:503] Master 127.21.28.126:46061 was elected leader, sending a full tablet report...
I20260808 07:00:12.778884 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:12.771719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.21.28.65:41275)
0808 07:00:12.775758 (+ 4039us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:12.775761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:12.775771 (+ 10us) server_negotiation.cc:407] Connection header received
0808 07:00:12.775796 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:12.775798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:12.775809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:12.775872 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:12.776044 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.776669 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.778139 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.778572 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.778624 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:12.778657 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:12.778812 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 07:00:12.778838 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4015,"thread_start_us":91,"threads_started":1}
I20260808 07:00:12.778923 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:12.771632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41275 (local address 127.0.0.1:33940)
0808 07:00:12.771874 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:12.771907 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:12.771973 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:12.775894 (+ 3921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:12.775897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:12.775900 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:12.775963 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:12.775965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.776728 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.776729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:12.778062 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:12.778066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.778637 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.778638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:12.778723 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:12.778736 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 07:00:12.778794 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 07:00:12.778843 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
I20260808 07:00:12.779313 13372 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5dff56143b4494a16e5df6c0f3be6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:59:52 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:33940
I20260808 07:00:12.779786 13509 tablet_replica.cc:330] stopping tablet replica
I20260808 07:00:12.779876 13509 raft_consensus.cc:2229] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:12.779942 13509 raft_consensus.cc:2258] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:12.780217 13509 ts_tablet_manager.cc:1905] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:12.782418 13509 ts_tablet_manager.cc:1918] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:12.782477 13509 log.cc:1196] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal/wals/5d5dff56143b4494a16e5df6c0f3be6e
I20260808 07:00:12.782684 13509 ts_tablet_manager.cc:1939] T 5d5dff56143b4494a16e5df6c0f3be6e P f23559b7d9904310ac964cd824675129: Deleting consensus metadata
I20260808 07:00:12.783118 13184 catalog_manager.cc:4841] TS f23559b7d9904310ac964cd824675129 (127.21.28.65:41275): tablet 5d5dff56143b4494a16e5df6c0f3be6e (table test-table [id=e067b80e9e674ddf9abebf74c39f9fd7]) successfully deleted
W20260808 07:00:12.843627 13499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:12.843873 13499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:12.843904 13499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:12.845019 13499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:12.845057 13499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:12.845090 13499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 07:00:12.846463 13499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:40827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=45357
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:12.846676 13499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:12.846900 13499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:12.851622 13529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:12.851655 13514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:12.851617 13515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:12.851837 13499 server_base.cc:1034] running on GCE node
I20260808 07:00:12.852035 13499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:12.852219 13499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:12.853359 13499 hybrid_clock.cc:648] HybridClock initialized: now 1786172412853342 us; error 31 us; skew 500 ppm
Aug 08 07:00:12 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:12.855813 13499 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 07:00:12.856202 13499 webserver.cc:460] Webserver started at http://127.21.28.66:45357/ using document root <none> and password file <none>
I20260808 07:00:12.856393 13499 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:12.856455 13499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:12.857805 13499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:12.858367 13538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:12.858517 13499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:12.858599 13499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "a1927b504e0d4f1496430f95b5e6d3c4"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "7ef033d4bcbbbfeec03ae4ff8f472e2a"
server_key_iv: "aa293a659feb65fdc9c1b3061b379806"
server_key_version: "encryptionkey@0"
I20260808 07:00:12.858928 13499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:12.877029 13499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:12.878999 13499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:12.879129 13499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:12.879331 13499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:12.879616 13499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:12.879647 13499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:12.879694 13499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:12.879717 13499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:12.885692 13499 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40827
I20260808 07:00:12.885762 13714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40827 every 8 connection(s)
I20260808 07:00:12.886047 13499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 07:00:12.887327 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 13499
I20260808 07:00:12.887588 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:38411
--local_ip_for_outbound_sockets=127.21.28.67
--tserver_master_addrs=127.21.28.126:46061
--webserver_port=44663
--webserver_interface=127.21.28.67
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 07:00:12 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:12.894969 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:12.886701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:43793 (local address 127.21.28.126:46061)
0808 07:00:12.886757 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:12.886759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:12.886976 (+ 217us) server_negotiation.cc:407] Connection header received
0808 07:00:12.887489 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:12.887491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:12.887507 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:12.887558 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:12.888515 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.889020 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.889971 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.890185 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.890240 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:12.892350 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:12.892352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:12.892352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:12.892360 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:12.893757 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:12.894112 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:12.894113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:12.894113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:12.894175 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:12.894370 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:12.894371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:12.894372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:12.894508 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:12.894578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:12.894776 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 07:00:12.894872 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 07:00:12.895018 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:12.886671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:43793)
0808 07:00:12.886870 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:12.886903 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:12.887377 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:12.887625 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:12.887629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:12.888220 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:12.888420 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:12.888425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.889102 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.889103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:12.889890 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:12.889893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:12.890252 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:12.890253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:12.890374 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:12.890784 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:12.890792 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:12.892248 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:12.893841 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:12.893847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:12.893848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:12.894047 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:12.894233 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:12.894233 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:12.894234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:12.894318 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:12.894566 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:12.894569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:12.894668 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 07:00:12.894738 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 07:00:12.894867 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":118,"threads_started":1}
I20260808 07:00:12.895457 13715 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46061
I20260808 07:00:12.895560 13715 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:12.895788 13715 heartbeater.cc:511] Master 127.21.28.126:46061 requested a full tablet report, sending...
I20260808 07:00:12.896158 13210 ts_manager.cc:194] Registered new tserver with Master: a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827)
I20260808 07:00:12.896507 13210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:43793
W20260808 07:00:12.969699 13721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:12.969883 13721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:12.969905 13721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:12.971009 13721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:12.971050 13721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:12.971074 13721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 07:00:12.972528 13721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:38411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=44663
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:12.972743 13721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:12.972941 13721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:12.977543 13730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:12.977573 13727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:12.977552 13728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:12.977787 13721 server_base.cc:1034] running on GCE node
I20260808 07:00:12.977972 13721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:12.978153 13721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:12.979282 13721 hybrid_clock.cc:648] HybridClock initialized: now 1786172412979277 us; error 31 us; skew 500 ppm
Aug 08 07:00:12 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:12.981623 13721 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 07:00:12.982023 13721 webserver.cc:460] Webserver started at http://127.21.28.67:44663/ using document root <none> and password file <none>
I20260808 07:00:12.982218 13721 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:12.982276 13721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:12.983497 13721 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:12.984058 13737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:12.984193 13721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:12.984279 13721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "16bb79f2c5fd43ec823f25fdd0814077"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "5efa1dd023337ad501e27d04b34a3f03"
server_key_iv: "503434ad3d092e8c8710096c2071bedb"
server_key_version: "encryptionkey@0"
I20260808 07:00:12.984565 13721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:12.998509 13721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:13.000142 13721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:13.000268 13721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:13.000463 13721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:13.000728 13721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:13.000761 13721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:13.000779 13721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:13.000802 13721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:13.006919 13721 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:38411
I20260808 07:00:13.006961 13917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:38411 every 8 connection(s)
I20260808 07:00:13.007273 13721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 07:00:13 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:13.012888 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 13721
I20260808 07:00:13.016281 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.007920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:35373 (local address 127.21.28.126:46061)
0808 07:00:13.007965 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.007967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.008153 (+ 186us) server_negotiation.cc:407] Connection header received
0808 07:00:13.008623 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.008625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.008639 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.008689 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:13.009573 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.010092 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.011066 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.011272 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.011326 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.013460 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:13.013461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:13.013462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:13.013469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:13.014894 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:13.015293 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:13.015294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:13.015295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:13.015354 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:13.015585 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:13.015586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:13.015586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:13.015762 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:13.015823 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:13.016092 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 07:00:13.016181 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 07:00:13.016431 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.007907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:35373)
0808 07:00:13.008037 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:13.008068 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:13.008520 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:13.008750 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:13.008755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:13.009289 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:13.009478 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.009482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.010177 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.010178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.010977 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.010980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.011340 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.011341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.011460 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.011881 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:13.011888 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:13.013328 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:13.014979 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:13.014985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:13.014987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:13.015220 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:13.015414 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:13.015415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:13.015416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:13.015524 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:13.015837 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:13.015844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:13.015969 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 07:00:13.016062 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 07:00:13.016225 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260808 07:00:13.016877 13919 heartbeater.cc:347] Connected to a master server at 127.21.28.126:46061
I20260808 07:00:13.016963 13919 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:13.017138 13919 heartbeater.cc:511] Master 127.21.28.126:46061 requested a full tablet report, sending...
I20260808 07:00:13.017549 13210 ts_manager.cc:194] Registered new tserver with Master: 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411)
I20260808 07:00:13.018013 13210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:35373
I20260808 07:00:13.019368 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.013304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.21.28.126:46061)
0808 07:00:13.013419 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.013421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.013466 (+ 45us) server_negotiation.cc:407] Connection header received
0808 07:00:13.013529 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.013529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.013541 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.013600 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:13.014136 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.014597 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.015306 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.015504 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.015555 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.016824 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:13.016825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:13.016825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:13.016832 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:13.018347 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:13.018739 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:13.018740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:13.018740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:13.018793 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:13.018973 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:13.018974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:13.018974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:13.019069 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:13.019195 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:13.019225 (+ 30us) server_negotiation.cc:299] Negotiation successful
0808 07:00:13.019290 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260808 07:00:13.020051 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:00:13.022707 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.020372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.21.28.126:46061)
0808 07:00:13.020410 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.020412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.020513 (+ 101us) server_negotiation.cc:407] Connection header received
0808 07:00:13.020590 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.020590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.020624 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.020688 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:00:13.020805 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.021300 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.022030 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.022239 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.022299 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.022448 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:00:13.022492 (+ 44us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 07:00:13.022635 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36644: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 07:00:13.022765 13924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36644: token expired
W20260808 07:00:13.022924 13110 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.28.126:46061)
I20260808 07:00:13.028309 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.023148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.21.28.126:46061)
0808 07:00:13.023180 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.023182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.023189 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:00:13.023252 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.023253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.023262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.023318 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:13.023786 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.024259 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.024926 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.025159 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.025228 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.026172 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:13.026173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:13.026174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:13.026180 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:13.027443 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:13.027760 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:13.027761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:13.027761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:13.027811 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:13.027945 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:13.027945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:13.027945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:13.028051 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:13.028122 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:13.028197 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 07:00:13.028255 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 07:00:13.029357 13210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:00:13.030921 13210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:00:13.032986 13372 tablet_service.cc:1467] Processing CreateTablet for tablet b62588b8c6894cee858eeaa0b0707bad (DEFAULT_TABLE table=test-table [id=b0d686c6d9814094ab2e19e6308f8f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:13.033201 13372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b62588b8c6894cee858eeaa0b0707bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:13.034884 13324 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Bootstrap starting.
I20260808 07:00:13.035221 13324 tablet_bootstrap.cc:654] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:13.035813 13324 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: No bootstrap required, opened a new log
I20260808 07:00:13.035864 13324 ts_tablet_manager.cc:1397] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:13.035971 13324 raft_consensus.cc:348] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.036036 13324 raft_consensus.cc:374] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:13.036087 13324 raft_consensus.cc:729] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23559b7d9904310ac964cd824675129, State: Initialized, Role: FOLLOWER
I20260808 07:00:13.036134 13324 consensus_queue.cc:260] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.036230 13324 ts_tablet_manager.cc:1428] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:13.036667 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.032807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40827 (local address 127.0.0.1:40494)
0808 07:00:13.032970 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:13.032982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:13.033035 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:13.033370 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:13.033371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:13.033373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:13.033429 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.033431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.034607 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.034608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.035556 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.035559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.036440 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.036441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.036499 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.036503 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:13.036563 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 07:00:13.036600 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":12,"thread_start_us":78,"threads_started":1}
I20260808 07:00:13.036695 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.033023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.21.28.66:40827)
0808 07:00:13.033221 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.033225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.033235 (+ 10us) server_negotiation.cc:407] Connection header received
0808 07:00:13.033274 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.033276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.033291 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.033367 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:13.033497 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.034538 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.035652 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.036371 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.036442 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.036497 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:13.036578 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 07:00:13.036619 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260808 07:00:13.037173 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.032684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:38411 (local address 127.0.0.1:43858)
0808 07:00:13.032729 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:13.032740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:13.032778 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:13.033141 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:13.033142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:13.033144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:13.033183 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.033184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.034481 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.034482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.035432 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.035435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.036990 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.036991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.037046 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.037049 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 07:00:13.037096 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 07:00:13.037131 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 07:00:13.037209 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.032771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.21.28.67:38411)
0808 07:00:13.032978 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.032982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.032992 (+ 10us) server_negotiation.cc:407] Connection header received
0808 07:00:13.033024 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.033026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.033040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.033271 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:13.033341 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.034389 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.035519 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.036891 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.036988 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.037042 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:13.037105 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 07:00:13.037138 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":110,"threads_started":1}
I20260808 07:00:13.037339 13599 tablet_service.cc:1467] Processing CreateTablet for tablet b62588b8c6894cee858eeaa0b0707bad (DEFAULT_TABLE table=test-table [id=b0d686c6d9814094ab2e19e6308f8f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:13.037653 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b62588b8c6894cee858eeaa0b0707bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:13.037864 13814 tablet_service.cc:1467] Processing CreateTablet for tablet b62588b8c6894cee858eeaa0b0707bad (DEFAULT_TABLE table=test-table [id=b0d686c6d9814094ab2e19e6308f8f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:13.038172 13814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b62588b8c6894cee858eeaa0b0707bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:13.041182 13939 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Bootstrap starting.
I20260808 07:00:13.041467 13940 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Bootstrap starting.
I20260808 07:00:13.041774 13939 tablet_bootstrap.cc:654] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:13.042026 13940 tablet_bootstrap.cc:654] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:13.042255 13939 log.cc:824] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:13.042516 13940 log.cc:824] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:13.042969 13939 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: No bootstrap required, opened a new log
I20260808 07:00:13.043037 13939 ts_tablet_manager.cc:1397] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:13.043268 13940 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: No bootstrap required, opened a new log
I20260808 07:00:13.043320 13940 ts_tablet_manager.cc:1397] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:13.044503 13939 raft_consensus.cc:348] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.044651 13939 raft_consensus.cc:374] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:13.044692 13939 raft_consensus.cc:729] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1927b504e0d4f1496430f95b5e6d3c4, State: Initialized, Role: FOLLOWER
I20260808 07:00:13.044801 13939 consensus_queue.cc:260] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.044782 13940 raft_consensus.cc:348] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.044942 13940 raft_consensus.cc:374] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:13.044986 13940 raft_consensus.cc:729] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bb79f2c5fd43ec823f25fdd0814077, State: Initialized, Role: FOLLOWER
I20260808 07:00:13.045058 13715 heartbeater.cc:503] Master 127.21.28.126:46061 was elected leader, sending a full tablet report...
I20260808 07:00:13.045097 13940 consensus_queue.cc:260] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.045046 13939 ts_tablet_manager.cc:1428] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 07:00:13.045372 13940 ts_tablet_manager.cc:1428] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:13.045486 13919 heartbeater.cc:503] Master 127.21.28.126:46061 was elected leader, sending a full tablet report...
W20260808 07:00:13.136551 13716 tablet.cc:2369] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:13.254835 13947 raft_consensus.cc:482] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:13.254984 13947 raft_consensus.cc:504] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.255350 13947 leader_election.cc:288] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827), f23559b7d9904310ac964cd824675129 (127.21.28.65:41275)
W20260808 07:00:13.256387 13494 tablet.cc:2369] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:00:13.257776 13920 tablet.cc:2369] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:13.258478 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.255543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41275 (local address 127.21.28.67:60189)
0808 07:00:13.255723 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:13.255731 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:13.255791 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:13.255920 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:13.255921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:13.255923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:13.255974 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.255975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.256840 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.256842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.257646 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.257648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.258227 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.258228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.258300 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.258306 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 07:00:13.258365 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 07:00:13.258409 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
I20260808 07:00:13.258518 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.255590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:60189 (local address 127.21.28.65:41275)
0808 07:00:13.255629 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.255630 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.255802 (+ 172us) server_negotiation.cc:407] Connection header received
0808 07:00:13.255860 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.255861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.255869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.255908 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:13.256026 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.256734 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.257728 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.258153 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.258211 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.258219 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:13.258402 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 07:00:13.258453 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:00:13.258653 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.255525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40827 (local address 127.21.28.67:57063)
0808 07:00:13.255582 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:13.255594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:13.255638 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:13.255789 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:13.255790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:13.255792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:13.255864 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.255866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.256787 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.256789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.257669 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:13.257672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.258403 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.258404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:13.258486 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.258491 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 07:00:13.258553 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 07:00:13.258588 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 07:00:13.258903 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.255599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:57063 (local address 127.21.28.66:40827)
0808 07:00:13.255647 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.255649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.255660 (+ 11us) server_negotiation.cc:407] Connection header received
0808 07:00:13.255711 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.255712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.255719 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.255768 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:13.255925 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.256699 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.257782 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.258299 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.258375 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.258383 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:13.258803 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 07:00:13.258855 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 07:00:13.258848 13418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62588b8c6894cee858eeaa0b0707bad" candidate_uuid: "16bb79f2c5fd43ec823f25fdd0814077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23559b7d9904310ac964cd824675129" is_pre_election: true
I20260808 07:00:13.259001 13418 raft_consensus.cc:2454] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16bb79f2c5fd43ec823f25fdd0814077 in term 0.
I20260808 07:00:13.259184 13631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62588b8c6894cee858eeaa0b0707bad" candidate_uuid: "16bb79f2c5fd43ec823f25fdd0814077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" is_pre_election: true
I20260808 07:00:13.259217 13740 leader_election.cc:302] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bb79f2c5fd43ec823f25fdd0814077, f23559b7d9904310ac964cd824675129; no voters:
I20260808 07:00:13.259308 13631 raft_consensus.cc:2454] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16bb79f2c5fd43ec823f25fdd0814077 in term 0.
I20260808 07:00:13.259343 13947 raft_consensus.cc:2781] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:00:13.259392 13947 raft_consensus.cc:482] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:00:13.259414 13947 raft_consensus.cc:3037] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:13.260265 13947 raft_consensus.cc:504] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.260396 13947 leader_election.cc:288] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [CANDIDATE]: Term 1 election: Requested vote from peers a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827), f23559b7d9904310ac964cd824675129 (127.21.28.65:41275)
I20260808 07:00:13.260604 13418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62588b8c6894cee858eeaa0b0707bad" candidate_uuid: "16bb79f2c5fd43ec823f25fdd0814077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23559b7d9904310ac964cd824675129"
I20260808 07:00:13.260610 13631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62588b8c6894cee858eeaa0b0707bad" candidate_uuid: "16bb79f2c5fd43ec823f25fdd0814077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1927b504e0d4f1496430f95b5e6d3c4"
I20260808 07:00:13.260681 13418 raft_consensus.cc:3037] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:13.260706 13631 raft_consensus.cc:3037] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:13.261497 13631 raft_consensus.cc:2454] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16bb79f2c5fd43ec823f25fdd0814077 in term 1.
I20260808 07:00:13.261667 13741 leader_election.cc:302] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bb79f2c5fd43ec823f25fdd0814077, a1927b504e0d4f1496430f95b5e6d3c4; no voters:
I20260808 07:00:13.261796 13418 raft_consensus.cc:2454] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16bb79f2c5fd43ec823f25fdd0814077 in term 1.
I20260808 07:00:13.261776 13947 raft_consensus.cc:2781] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:13.261986 13947 raft_consensus.cc:686] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 LEADER]: Becoming Leader. State: Replica: 16bb79f2c5fd43ec823f25fdd0814077, State: Running, Role: LEADER
I20260808 07:00:13.262071 13947 consensus_queue.cc:237] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:13.262838 13208 catalog_manager.cc:5495] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67). New cstate: current_term: 1 leader_uuid: "16bb79f2c5fd43ec823f25fdd0814077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } health_report { overall_health: UNKNOWN } } }
I20260808 07:00:13.316991 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.314699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.21.28.67:38411)
0808 07:00:13.314763 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:13.314765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:13.314775 (+ 10us) server_negotiation.cc:407] Connection header received
0808 07:00:13.314818 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:13.314818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:13.314836 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:13.314893 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:00:13.315014 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.315481 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.316262 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:13.316462 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:13.316518 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:13.316666 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:00:13.316763 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:00:13.316835 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:13.316876 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 07:00:13.316895 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 07:00:13.318729 13631 raft_consensus.cc:1261] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 FOLLOWER]: Refusing update from remote peer 16bb79f2c5fd43ec823f25fdd0814077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:13.318729 13418 raft_consensus.cc:1261] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Refusing update from remote peer 16bb79f2c5fd43ec823f25fdd0814077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:13.319077 13947 consensus_queue.cc:1035] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:13.319216 13951 consensus_queue.cc:1035] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:13.321115 13954 mvcc.cc:205] Tried to move back new op lower bound from 7316162204950761472 to 7316162204721860608. Current Snapshot: MvccSnapshot[applied={T|T < 7316162204950761472}]
I20260808 07:00:13.321760 13957 mvcc.cc:205] Tried to move back new op lower bound from 7316162204950761472 to 7316162204721860608. Current Snapshot: MvccSnapshot[applied={T|T < 7316162204950761472}]
I20260808 07:00:13.322309 13955 mvcc.cc:205] Tried to move back new op lower bound from 7316162204950761472 to 7316162204721860608. Current Snapshot: MvccSnapshot[applied={T|T < 7316162204950761472 or (T in {7316162204950761472})}]
I20260808 07:00:13.323318 13208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:13.323395 13208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:13.324059 13208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Sending DeleteTablet for 3 replicas of tablet b62588b8c6894cee858eeaa0b0707bad
I20260808 07:00:13.324313 13814 tablet_service.cc:1514] Processing DeleteTablet for tablet b62588b8c6894cee858eeaa0b0707bad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:13 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:43858
I20260808 07:00:13.324378 13599 tablet_service.cc:1514] Processing DeleteTablet for tablet b62588b8c6894cee858eeaa0b0707bad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:13 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:40494
I20260808 07:00:13.324388 13372 tablet_service.cc:1514] Processing DeleteTablet for tablet b62588b8c6894cee858eeaa0b0707bad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:13 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:33940
I20260808 07:00:13.324443 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 13264
I20260808 07:00:13.329200 13969 tablet_replica.cc:330] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: stopping tablet replica
I20260808 07:00:13.329356 13969 raft_consensus.cc:2229] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:13.329515 13969 raft_consensus.cc:2258] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:13.330446 13969 ts_tablet_manager.cc:1905] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:13.332540 13969 ts_tablet_manager.cc:1918] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:13.332623 13969 log.cc:1196] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/b62588b8c6894cee858eeaa0b0707bad
I20260808 07:00:13.332917 13969 ts_tablet_manager.cc:1939] T b62588b8c6894cee858eeaa0b0707bad P a1927b504e0d4f1496430f95b5e6d3c4: Deleting consensus metadata
I20260808 07:00:13.333334 13187 catalog_manager.cc:4841] TS a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827): tablet b62588b8c6894cee858eeaa0b0707bad (table test-table [id=b0d686c6d9814094ab2e19e6308f8f63]) successfully deleted
W20260808 07:00:13.338732 13184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:41275 (error 108)
I20260808 07:00:13.338944 13968 tablet_replica.cc:330] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: stopping tablet replica
I20260808 07:00:13.339033 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 13499
I20260808 07:00:13.339058 13968 raft_consensus.cc:2229] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:00:13.339155 13968 raft_consensus.cc:2258] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:13.341125 13968 ts_tablet_manager.cc:1905] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:13.352075 13968 ts_tablet_manager.cc:1918] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:13.352157 13968 log.cc:1196] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/b62588b8c6894cee858eeaa0b0707bad
I20260808 07:00:13.352435 13968 ts_tablet_manager.cc:1939] T b62588b8c6894cee858eeaa0b0707bad P 16bb79f2c5fd43ec823f25fdd0814077: Deleting consensus metadata
I20260808 07:00:13.352610 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:13.352449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:41275 (local address 127.0.0.1:33952)
0808 07:00:13.352515 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:13.352572 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 07:00:13.352923 13187 catalog_manager.cc:4841] TS 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411): tablet b62588b8c6894cee858eeaa0b0707bad (table test-table [id=b0d686c6d9814094ab2e19e6308f8f63]) successfully deleted
W20260808 07:00:13.353521 13184 catalog_manager.cc:4584] TS f23559b7d9904310ac964cd824675129 (127.21.28.65:41275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b62588b8c6894cee858eeaa0b0707bad: Network error: Client connection negotiation failed: client connection to 127.21.28.65:41275: connect: Connection refused (error 111)
I20260808 07:00:13.356765 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 13721
I20260808 07:00:13.368346 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 13164
I20260808 07:00:33.375002 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46061
--webserver_interface=127.21.28.126
--webserver_port=42583
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:33.455884 13971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:33.456066 13971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:33.456095 13971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:33.457237 13971 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 07:00:33.457276 13971 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 07:00:33.457293 13971 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 07:00:33.457306 13971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:33.457324 13971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:33.457347 13971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:33.457370 13971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:33.457392 13971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:33.458891 13971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:33.459117 13971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:33.459317 13971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:33.464077 13977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:33.464036 13979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:33.464089 13971 server_base.cc:1034] running on GCE node
W20260808 07:00:33.464044 13976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:33.464521 13971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:33.464841 13971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:33.465986 13971 hybrid_clock.cc:648] HybridClock initialized: now 1786172433465971 us; error 28 us; skew 500 ppm
Aug 08 07:00:33 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:33.468377 13971 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:00:33.468773 13971 webserver.cc:460] Webserver started at http://127.21.28.126:42583/ using document root <none> and password file <none>
I20260808 07:00:33.468978 13971 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:33.469022 13971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:33.470220 13971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:33.470780 13988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.470940 13971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:33.471031 13971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "439d11b42cc4478fabfec020ce227ba9"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "408387d284995525093fa932c92fcab9"
server_key_iv: "881454d2091c9c59306dd7f168ad62c5"
server_key_version: "encryptionkey@0"
I20260808 07:00:33.471307 13971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:33.496628 13971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:33.498559 13971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:33.498668 13971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:33.502916 13971 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46061
I20260808 07:00:33.502961 14083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46061 every 8 connection(s)
I20260808 07:00:33.503362 13971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:33.504971 14084 sys_catalog.cc:263] Verifying existing consensus state
I20260808 07:00:33.505815 14084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Bootstrap starting.
I20260808 07:00:33.510915 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 13971
I20260808 07:00:33.511206 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:41275
--local_ip_for_outbound_sockets=127.21.28.65
--tserver_master_addrs=127.21.28.126:46061
--webserver_port=46499
--webserver_interface=127.21.28.65
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 07:00:33.516098 14084 log.cc:824] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:33.518198 14084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 07:00:33.518492 14084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Bootstrap complete.
I20260808 07:00:33.520562 14084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 07:00:33.520782 14084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439d11b42cc4478fabfec020ce227ba9, State: Initialized, Role: FOLLOWER
I20260808 07:00:33.520923 14084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 07:00:33.520985 14084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:33.521026 14084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:33.521075 14084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 07:00:33.522037 14084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 07:00:33.522133 14084 leader_election.cc:302] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439d11b42cc4478fabfec020ce227ba9; no voters:
I20260808 07:00:33.522284 14084 leader_election.cc:288] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 07:00:33.522329 14090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 07:00:33.522455 14090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [term 3 LEADER]: Becoming Leader. State: Replica: 439d11b42cc4478fabfec020ce227ba9, State: Running, Role: LEADER
I20260808 07:00:33.522538 14084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:33.522538 14090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } }
I20260808 07:00:33.523111 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439d11b42cc4478fabfec020ce227ba9. Latest consensus state: current_term: 3 leader_uuid: "439d11b42cc4478fabfec020ce227ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } } }
I20260808 07:00:33.523211 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:33.523119 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "439d11b42cc4478fabfec020ce227ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439d11b42cc4478fabfec020ce227ba9" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46061 } } }
I20260808 07:00:33.523355 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:33.523785 14107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:33.524451 14107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:33.524545 14107 catalog_manager.cc:1259] Loaded cluster ID: 75ee1e815494439ca0d2d1ef39d79ba5
I20260808 07:00:33.524580 14107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:00:33.525382 14107 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:33.525552 14107 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Loaded TSK: 0
I20260808 07:00:33.525596 14107 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 439d11b42cc4478fabfec020ce227ba9: Loaded TSK: 1
F20260808 07:00:33.525836 14107 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f600fc369ad google::LogMessage::Fail() at ??:0
@ 0x7f600fc3988a google::LogMessage::SendToLog() at ??:0
@ 0x7f600fc364de google::LogMessage::Flush() at ??:0
@ 0x7f600fc37a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f60123a3104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f60123a0d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6017b901cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6017b9048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6017b90594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6017b56c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6017b60ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6011b040d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6011af9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f601295d6db start_thread at ??:0
@ 0x7f600e73461f clone at ??:0
W20260808 07:00:33.606128 14088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:33.606321 14088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:33.606354 14088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:33.607491 14088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:33.607532 14088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:33.607568 14088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 07:00:33.609006 14088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:41275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=46499
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:33.609239 14088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:33.609457 14088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:33.614720 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:33.614687 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:33.614917 14088 server_base.cc:1034] running on GCE node
W20260808 07:00:33.614691 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:33.615252 14088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:33.615518 14088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:33.616699 14088 hybrid_clock.cc:648] HybridClock initialized: now 1786172433616679 us; error 34 us; skew 500 ppm
Aug 08 07:00:33 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:33.619791 14088 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 07:00:33.620345 14088 webserver.cc:460] Webserver started at http://127.21.28.65:46499/ using document root <none> and password file <none>
I20260808 07:00:33.620604 14088 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:33.620674 14088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:33.622349 14088 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:33.623101 14164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.623272 14088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:33.623375 14088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "f23559b7d9904310ac964cd824675129"
format_stamp: "Formatted at 2026-08-08 06:59:51 on dist-test-slave-nj21"
server_key: "2c54bfef8b4e1b2a505e9c12cf8ff8c9"
server_key_iv: "b6449189da075007d5a5d24ed6dbbbdd"
server_key_version: "encryptionkey@0"
I20260808 07:00:33.623780 14088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:33.638260 14088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:33.640589 14088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:33.640733 14088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:33.640983 14088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:33.641460 14176 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 07:00:33.642810 14088 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 07:00:33.642872 14088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 07:00:33.642902 14088 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 07:00:33.643807 14088 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 07:00:33.643854 14088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:33.643899 14176 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Bootstrap starting.
I20260808 07:00:33.650996 14088 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:41275
I20260808 07:00:33.651158 14347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:41275 every 8 connection(s)
I20260808 07:00:33.651448 14088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 07:00:33.653781 14176 log.cc:824] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:33.656534 14176 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 07:00:33.656852 14176 tablet_bootstrap.cc:492] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Bootstrap complete.
I20260808 07:00:33.657117 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 14088
I20260808 07:00:33.657362 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:40827
--local_ip_for_outbound_sockets=127.21.28.66
--tserver_master_addrs=127.21.28.126:46061
--webserver_port=45357
--webserver_interface=127.21.28.66
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 07:00:33.657932 14176 ts_tablet_manager.cc:1397] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260808 07:00:33.659264 14176 raft_consensus.cc:348] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:33.659595 14176 raft_consensus.cc:729] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23559b7d9904310ac964cd824675129, State: Initialized, Role: FOLLOWER
I20260808 07:00:33.659757 14176 consensus_queue.cc:260] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:33.660034 14176 ts_tablet_manager.cc:1428] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260808 07:00:33.724857 14354 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:00:33.652211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:58005)
0808 07:00:33.652436 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.652488 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:33.653133 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:33.724627 (+ 71494us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":106,"threads_started":1}
W20260808 07:00:33.725116 14349 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:46061 (0 consecutive failures): Network error: Failed to ping master at 127.21.28.126:46061: Client connection negotiation failed: client connection to 127.21.28.126:46061: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 07:00:33.725471 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:33.725352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:41645)
0808 07:00:33.725397 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.725448 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 07:00:33.725819 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:33.725743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:59359)
0808 07:00:33.725782 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.725805 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 07:00:33.725925 14349 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 07:00:33.744537 14361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:33.744709 14361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:33.744730 14361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:33.745800 14361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:33.745832 14361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:33.745855 14361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 07:00:33.747216 14361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:40827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=45357
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:33.747397 14361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:33.747573 14361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:33.752283 14372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:33.752298 14369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:33.752482 14361 server_base.cc:1034] running on GCE node
W20260808 07:00:33.752300 14368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:33.752753 14361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:33.752956 14361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:33.754094 14361 hybrid_clock.cc:648] HybridClock initialized: now 1786172433754077 us; error 35 us; skew 500 ppm
Aug 08 07:00:33 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:33.756457 14361 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 07:00:33.756901 14361 webserver.cc:460] Webserver started at http://127.21.28.66:45357/ using document root <none> and password file <none>
I20260808 07:00:33.757115 14361 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:33.757159 14361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:33.758368 14361 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:33.758874 14384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.758999 14361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.759074 14361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "a1927b504e0d4f1496430f95b5e6d3c4"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "7ef033d4bcbbbfeec03ae4ff8f472e2a"
server_key_iv: "aa293a659feb65fdc9c1b3061b379806"
server_key_version: "encryptionkey@0"
I20260808 07:00:33.759361 14361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:33.792675 14361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:33.794390 14361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:33.794526 14361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:33.794731 14361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:33.795022 14361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:33.795054 14361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.795076 14361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:33.795099 14361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.800535 14361 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40827
I20260808 07:00:33.800603 14557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40827 every 8 connection(s)
I20260808 07:00:33.800887 14361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 07:00:33.801838 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:33.801498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:56395)
0808 07:00:33.801638 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.801703 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
W20260808 07:00:33.802201 14559 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:46061 (0 consecutive failures): Network error: Failed to ping master at 127.21.28.126:46061: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
I20260808 07:00:33.802430 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:33.802347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:42341)
0808 07:00:33.802388 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.802413 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 07:00:33.802634 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:33.802575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:34349)
0808 07:00:33.802605 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.802622 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260808 07:00:33.802712 14559 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 07:00:33.803772 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 14361
I20260808 07:00:33.804037 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:38411
--local_ip_for_outbound_sockets=127.21.28.67
--tserver_master_addrs=127.21.28.126:46061
--webserver_port=44663
--webserver_interface=127.21.28.67
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:33.883507 14567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:33.883724 14567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:33.883756 14567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:33.884868 14567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:33.884909 14567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:33.884953 14567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 07:00:33.886374 14567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:38411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=44663
--tserver_master_addrs=127.21.28.126:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:33.886612 14567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:33.886813 14567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:33.891564 14579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:33.891597 14567 server_base.cc:1034] running on GCE node
W20260808 07:00:33.891603 14576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:33.891561 14582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:33.891957 14567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:33.892184 14567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:33.893323 14567 hybrid_clock.cc:648] HybridClock initialized: now 1786172433893304 us; error 32 us; skew 500 ppm
Aug 08 07:00:33 dist-test-slave-nj21 krb5kdc[12328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:33.895601 14567 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 07:00:33.896037 14567 webserver.cc:460] Webserver started at http://127.21.28.67:44663/ using document root <none> and password file <none>
I20260808 07:00:33.896251 14567 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:33.896299 14567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:33.897560 14567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:00:33.898077 14598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.898226 14567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:33.898308 14567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "16bb79f2c5fd43ec823f25fdd0814077"
format_stamp: "Formatted at 2026-08-08 06:59:52 on dist-test-slave-nj21"
server_key: "5efa1dd023337ad501e27d04b34a3f03"
server_key_iv: "503434ad3d092e8c8710096c2071bedb"
server_key_version: "encryptionkey@0"
I20260808 07:00:33.898617 14567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:33.905604 14567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:33.907570 14567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:33.907732 14567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:33.907914 14567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:33.908169 14567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:33.908197 14567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.908254 14567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:33.908280 14567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:33.913957 14567 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:38411
I20260808 07:00:33.914018 14780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:38411 every 8 connection(s)
I20260808 07:00:33.914296 14567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 07:00:33.915167 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:33.914832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:45381)
0808 07:00:33.914978 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.915040 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
W20260808 07:00:33.915521 14781 heartbeater.cc:644] Failed to heartbeat to 127.21.28.126:46061 (0 consecutive failures): Network error: Failed to ping master at 127.21.28.126:46061: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
I20260808 07:00:33.915772 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:33.915690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:59473)
0808 07:00:33.915728 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.915755 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260808 07:00:33.915988 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:33.915925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:43105)
0808 07:00:33.915961 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:33.915978 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 07:00:33.916064 14781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 07:00:33.918774 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 14567
I20260808 07:00:34.726500 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:34.726273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:38707)
0808 07:00:34.726415 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:34.726449 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260808 07:00:34.803160 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:34.802977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:41453)
0808 07:00:34.803078 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:34.803112 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260808 07:00:34.916564 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:34.916355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:43899)
0808 07:00:34.916479 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:34.916510 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260808 07:00:34.985574 14792 raft_consensus.cc:482] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:34.985721 14792 raft_consensus.cc:504] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:34.986039 14792 leader_election.cc:288] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411), a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827)
Aug 08 07:00:34 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.66@KRBTEST.COM
Aug 08 07:00:34 dist-test-slave-nj21 krb5kdc[12328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.67@KRBTEST.COM
I20260808 07:00:34.999524 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:34.986244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:38411 (local address 127.21.28.65:39407)
0808 07:00:34.986305 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:34.986317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:34.986377 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:34.987285 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:34.987289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:34.987665 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:34.987911 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:34.987917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:34.988980 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:34.988981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:34.989802 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:34.989804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:34.990387 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:34.990388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:34.990509 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:34.994199 (+ 3690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:34.994204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:34.996164 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:34.997893 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:34.997894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:34.997894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:34.998169 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:34.998379 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:34.998379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:34.998379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:34.998476 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:34.998941 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:34.998950 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:34.999113 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 07:00:34.999232 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 07:00:34.999406 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":83}
I20260808 07:00:34.999552 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:34.986525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:57043 (local address 127.21.28.66:40827)
0808 07:00:34.986679 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:34.986690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:34.986730 (+ 40us) server_negotiation.cc:407] Connection header received
0808 07:00:34.987157 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:34.987159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:34.987205 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:34.987356 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:34.988021 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:34.988870 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:34.989914 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:34.990319 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:34.990429 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:34.995983 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:34.995988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:34.995989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:34.996001 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:34.997705 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:34.998401 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:34.998403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:34.998403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:34.998480 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:34.998648 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:34.998649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:34.998649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:34.998907 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:34.998975 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:34.999292 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 07:00:34.999434 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260808 07:00:34.999712 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:34.986368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40827 (local address 127.21.28.65:57043)
0808 07:00:34.986451 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:34.986459 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:34.986493 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:34.987339 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:34.987341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:34.987659 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:34.987911 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:34.987917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:34.988966 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:34.988967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:34.989796 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:34.989798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:34.990395 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:34.990396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:34.990506 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:34.994192 (+ 3686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:34.994201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:34.995869 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:34.997791 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:34.997798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:34.997800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:34.998169 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:34.998541 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:34.998541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:34.998542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:34.998596 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:34.998972 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:34.998973 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:34.999113 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 07:00:34.999232 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 07:00:34.999470 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":138,"thread_start_us":55,"threads_started":1}
I20260808 07:00:34.999549 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:34.986365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:39407 (local address 127.21.28.67:38411)
0808 07:00:34.986542 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:34.986552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:34.986582 (+ 30us) server_negotiation.cc:407] Connection header received
0808 07:00:34.987048 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:34.987050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:34.987092 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:34.987217 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:34.988032 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:34.988871 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:34.989887 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:34.990296 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:34.990395 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:34.996292 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:34.996296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:34.996296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:34.996306 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:34.997816 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:34.998237 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:34.998239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:34.998240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:34.998320 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:34.998547 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:34.998548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:34.998548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:34.998867 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:34.998955 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:34.999292 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 07:00:34.999431 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
W20260808 07:00:35.000473 14167 leader_election.cc:341] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827): Not found: Tablet not found: b62588b8c6894cee858eeaa0b0707bad
W20260808 07:00:35.000816 14167 leader_election.cc:341] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411): Not found: Tablet not found: b62588b8c6894cee858eeaa0b0707bad
I20260808 07:00:35.000916 14167 leader_election.cc:302] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f23559b7d9904310ac964cd824675129; no voters: 16bb79f2c5fd43ec823f25fdd0814077, a1927b504e0d4f1496430f95b5e6d3c4
I20260808 07:00:35.001042 14792 raft_consensus.cc:2726] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:00:35.727174 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:35.726977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:50403)
0808 07:00:35.727093 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:35.727129 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260808 07:00:35.803743 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:35.803524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:59519)
0808 07:00:35.803640 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:35.803699 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260808 07:00:35.917098 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:35.916925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:44363)
0808 07:00:35.917030 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:35.917063 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260808 07:00:36.727809 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:36.727555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:57695)
0808 07:00:36.727701 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:36.727751 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260808 07:00:36.804323 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:36.804095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:48123)
0808 07:00:36.804248 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:36.804286 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260808 07:00:36.855587 14810 raft_consensus.cc:482] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:36.855710 14810 raft_consensus.cc:504] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:36.855871 14810 leader_election.cc:288] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411), a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827)
W20260808 07:00:36.856271 14167 leader_election.cc:341] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827): Not found: Tablet not found: b62588b8c6894cee858eeaa0b0707bad
W20260808 07:00:36.856348 14167 leader_election.cc:341] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411): Not found: Tablet not found: b62588b8c6894cee858eeaa0b0707bad
I20260808 07:00:36.856370 14167 leader_election.cc:302] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f23559b7d9904310ac964cd824675129; no voters: 16bb79f2c5fd43ec823f25fdd0814077, a1927b504e0d4f1496430f95b5e6d3c4
I20260808 07:00:36.856473 14810 raft_consensus.cc:2726] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:00:36.917616 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:36.917436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:35741)
0808 07:00:36.917545 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:36.917578 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260808 07:00:37.728422 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:37.728197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:44947)
0808 07:00:37.728336 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:37.728375 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260808 07:00:37.804940 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:37.804751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:37683)
0808 07:00:37.804860 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:37.804894 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260808 07:00:37.918146 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:37.917949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:48711)
0808 07:00:37.918073 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:37.918108 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260808 07:00:38.729071 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:38.728864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:49793)
0808 07:00:38.728982 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:38.729018 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260808 07:00:38.805610 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:38.805387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:60053)
0808 07:00:38.805534 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:38.805569 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260808 07:00:38.918665 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:38.918474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:41299)
0808 07:00:38.918591 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:38.918625 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260808 07:00:39.505122 14825 raft_consensus.cc:482] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:39.505223 14825 raft_consensus.cc:504] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:39.505378 14825 leader_election.cc:288] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411), a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827)
W20260808 07:00:39.505690 14167 leader_election.cc:341] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411): Not found: Tablet not found: b62588b8c6894cee858eeaa0b0707bad
W20260808 07:00:39.505764 14167 leader_election.cc:341] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827): Not found: Tablet not found: b62588b8c6894cee858eeaa0b0707bad
I20260808 07:00:39.505782 14167 leader_election.cc:302] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f23559b7d9904310ac964cd824675129; no voters: 16bb79f2c5fd43ec823f25fdd0814077, a1927b504e0d4f1496430f95b5e6d3c4
I20260808 07:00:39.505851 14825 raft_consensus.cc:2726] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:00:39.729700 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:39.729489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:50471)
0808 07:00:39.729617 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:39.729652 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260808 07:00:39.806226 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:39.806047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:57415)
0808 07:00:39.806155 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:39.806186 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260808 07:00:39.919241 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:39.919058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:50363)
0808 07:00:39.919170 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:39.919202 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260808 07:00:40.730278 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:40.730068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:54683)
0808 07:00:40.730195 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:40.730231 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260808 07:00:40.806744 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:40.806565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:52895)
0808 07:00:40.806672 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:40.806704 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260808 07:00:40.919798 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:40.919595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:41961)
0808 07:00:40.919726 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:40.919758 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260808 07:00:41.730897 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:41.730684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:42507)
0808 07:00:41.730809 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:41.730849 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260808 07:00:41.807308 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:41.807099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:47391)
0808 07:00:41.807228 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:41.807269 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260808 07:00:41.920351 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:41.920170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:37161)
0808 07:00:41.920280 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:41.920311 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260808 07:00:42.731494 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:42.731290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:58567)
0808 07:00:42.731401 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:42.731446 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260808 07:00:42.807884 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:42.807699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:38377)
0808 07:00:42.807815 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:42.807847 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260808 07:00:42.920915 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:42.920719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:52461)
0808 07:00:42.920842 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:42.920873 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260808 07:00:43.732077 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:43.731886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:32821)
0808 07:00:43.731995 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:43.732031 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260808 07:00:43.808427 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:43.808244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:54079)
0808 07:00:43.808351 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:43.808381 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260808 07:00:43.857587 14846 raft_consensus.cc:482] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:43.857683 14846 raft_consensus.cc:504] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16bb79f2c5fd43ec823f25fdd0814077" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 38411 } } peers { permanent_uuid: "a1927b504e0d4f1496430f95b5e6d3c4" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40827 } } peers { permanent_uuid: "f23559b7d9904310ac964cd824675129" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 41275 } }
I20260808 07:00:43.857831 14846 leader_election.cc:288] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411), a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827)
W20260808 07:00:43.858206 14167 leader_election.cc:341] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bb79f2c5fd43ec823f25fdd0814077 (127.21.28.67:38411): Not found: Tablet not found: b62588b8c6894cee858eeaa0b0707bad
W20260808 07:00:43.858283 14167 leader_election.cc:341] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a1927b504e0d4f1496430f95b5e6d3c4 (127.21.28.66:40827): Not found: Tablet not found: b62588b8c6894cee858eeaa0b0707bad
I20260808 07:00:43.858311 14167 leader_election.cc:302] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f23559b7d9904310ac964cd824675129; no voters: 16bb79f2c5fd43ec823f25fdd0814077, a1927b504e0d4f1496430f95b5e6d3c4
I20260808 07:00:43.858390 14846 raft_consensus.cc:2726] T b62588b8c6894cee858eeaa0b0707bad P f23559b7d9904310ac964cd824675129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 07:00:43.921447 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:43.921251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:48169)
0808 07:00:43.921353 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:43.921388 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260808 07:00:44.732752 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:44.732499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:39599)
0808 07:00:44.732645 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:44.732689 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260808 07:00:44.808957 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:44.808757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:37053)
0808 07:00:44.808878 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:44.808907 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260808 07:00:44.921945 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:44.921771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:42323)
0808 07:00:44.921873 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:44.921904 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260808 07:00:45.733351 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:45.733144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:36591)
0808 07:00:45.733259 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:45.733300 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260808 07:00:45.809480 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:45.809279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:59809)
0808 07:00:45.809401 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:45.809429 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260808 07:00:45.922477 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:45.922291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:40413)
0808 07:00:45.922398 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:45.922432 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260808 07:00:46.733964 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:46.733746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:54053)
0808 07:00:46.733860 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:46.733918 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260808 07:00:46.810050 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:46.809854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:52975)
0808 07:00:46.809981 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:46.810011 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260808 07:00:46.923004 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:46.922813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:46807)
0808 07:00:46.922927 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:46.922960 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260808 07:00:47.734584 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:47.734356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:34363)
0808 07:00:47.734496 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:47.734533 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260808 07:00:47.810602 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:47.810419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:51141)
0808 07:00:47.810531 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:47.810562 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260808 07:00:47.923574 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:47.923368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:35359)
0808 07:00:47.923496 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:47.923529 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260808 07:00:48.735267 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:48.735057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.65:40051)
0808 07:00:48.735185 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:48.735221 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260808 07:00:48.811195 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:48.811017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.66:58177)
0808 07:00:48.811126 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:48.811155 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260808 07:00:48.923470 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 14088
I20260808 07:00:48.931483 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:48.931269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:46061 (local address 127.21.28.67:36203)
0808 07:00:48.931388 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:48.931431 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.126:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260808 07:00:48.935153 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 14361
I20260808 07:00:48.943656 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 14567
2026-08-08T07:00:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58541 ms)
[----------] 30 tests from SecurityITest (230172 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:00:48 dist-test-slave-nj21 krb5kdc[14875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:00:48 dist-test-slave-nj21 krb5kdc[14875](info): set up 2 sockets
Aug 08 07:00:48 dist-test-slave-nj21 krb5kdc[14875](info): commencing operation
krb5kdc: starting...
W20260808 07:00:50.100898 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:00:50.209431 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:43069
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:50.288748 14900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:50.288949 14900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:50.288981 14900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:50.290094 14900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:50.290133 14900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:50.290153 14900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:50.290169 14900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:50.290185 14900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:50.291599 14900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:50.291846 14900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:50.292081 14900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:50.296608 14915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:50.296762 14900 server_base.cc:1034] running on GCE node
W20260808 07:00:50.296703 14913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:50.296684 14921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:50.297124 14900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:50.297370 14900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:50.298502 14900 hybrid_clock.cc:648] HybridClock initialized: now 1786172450298492 us; error 32 us; skew 500 ppm
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:50.300806 14900 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:00:50.301204 14900 webserver.cc:460] Webserver started at http://127.21.28.126:44741/ using document root <none> and password file <none>
I20260808 07:00:50.301398 14900 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:50.301447 14900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:50.301565 14900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:50.302516 14900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "6f14d19098d34575b5274c296f78ac60"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "8c855695ff09cd7e4c8fbf73ba5774bb"
server_key_iv: "53f9b6c82c780075d3549a734a304b3d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.302841 14900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "6f14d19098d34575b5274c296f78ac60"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "8c855695ff09cd7e4c8fbf73ba5774bb"
server_key_iv: "53f9b6c82c780075d3549a734a304b3d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.304077 14900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:00:50.304716 14935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.304857 14900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:50.304940 14900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "6f14d19098d34575b5274c296f78ac60"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "8c855695ff09cd7e4c8fbf73ba5774bb"
server_key_iv: "53f9b6c82c780075d3549a734a304b3d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.304997 14900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:50.325351 14900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:50.327291 14900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:50.327427 14900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:50.331161 14900 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:43069
I20260808 07:00:50.331212 15022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:43069 every 8 connection(s)
I20260808 07:00:50.331523 14900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:50.332358 15026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:50.333264 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 14900
I20260808 07:00:50.333367 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:00:50.333552 21617 external_mini_cluster.cc:1428] Setting key a6af7cbfd523e75466a59559907d5e91
I20260808 07:00:50.334841 15026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60: Bootstrap starting.
I20260808 07:00:50.335573 15026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:50.335870 15026 log.cc:824] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:50.336496 15026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60: No bootstrap required, opened a new log
I20260808 07:00:50.338598 15026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f14d19098d34575b5274c296f78ac60" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 43069 } }
I20260808 07:00:50.338766 15026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:50.338815 15026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f14d19098d34575b5274c296f78ac60, State: Initialized, Role: FOLLOWER
I20260808 07:00:50.338945 15026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f14d19098d34575b5274c296f78ac60" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 43069 } }
I20260808 07:00:50.339027 15026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:50.339063 15026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:50.339114 15026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:50.339787 15026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f14d19098d34575b5274c296f78ac60" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 43069 } }
I20260808 07:00:50.339879 15026 leader_election.cc:302] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f14d19098d34575b5274c296f78ac60; no voters:
I20260808 07:00:50.340023 15026 leader_election.cc:288] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:00:50.340067 15033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 1 FOLLOWER]: Leader election won for term 1
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:50.340215 15033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [term 1 LEADER]: Becoming Leader. State: Replica: 6f14d19098d34575b5274c296f78ac60, State: Running, Role: LEADER
I20260808 07:00:50.340263 15026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:50.340379 15033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f14d19098d34575b5274c296f78ac60" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 43069 } }
I20260808 07:00:50.340956 15034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f14d19098d34575b5274c296f78ac60. Latest consensus state: current_term: 1 leader_uuid: "6f14d19098d34575b5274c296f78ac60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f14d19098d34575b5274c296f78ac60" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 43069 } } }
I20260808 07:00:50.341077 15034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:50.341385 15047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:50.340930 15035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f14d19098d34575b5274c296f78ac60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f14d19098d34575b5274c296f78ac60" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 43069 } } }
I20260808 07:00:50.341463 15035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:50.341882 15047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:50.343328 15047 catalog_manager.cc:1347] Generated new cluster ID: bf5753011c42438b985e91cd6998d6ea
I20260808 07:00:50.343380 15047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:00:50.344089 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.334025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.21.28.126:43069)
0808 07:00:50.334171 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:50.334179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:50.334205 (+ 26us) server_negotiation.cc:407] Connection header received
0808 07:00:50.334606 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:50.334611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:50.334654 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:50.334762 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:50.335517 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.336472 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.337511 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.337931 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.338008 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.340609 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:50.340612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:50.340613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:50.340624 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:50.342456 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.342777 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.342780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.342780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.342884 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.343098 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.343100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.343100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.343388 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:50.343506 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:50.343751 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 07:00:50.343890 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260808 07:00:50.349560 15047 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:00:50.350238 15047 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:50.353009 15047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60: Generated new TSK 0
I20260808 07:00:50.353154 15047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 07:00:50.378163 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:43069
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:50.457154 15073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:50.457346 15073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:50.457368 15073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:50.458473 15073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:50.458516 15073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:50.458541 15073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 07:00:50.459969 15073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:50.460163 15073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:50.460373 15073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:50.464985 15081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:50.465014 15085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:50.465018 15080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:50.465150 15073 server_base.cc:1034] running on GCE node
I20260808 07:00:50.465351 15073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:50.465550 15073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:50.466682 15073 hybrid_clock.cc:648] HybridClock initialized: now 1786172450466657 us; error 41 us; skew 500 ppm
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:50.469022 15073 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 07:00:50.469413 15073 webserver.cc:460] Webserver started at http://127.21.28.65:46845/ using document root <none> and password file <none>
I20260808 07:00:50.469590 15073 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:50.469630 15073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:50.469715 15073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:50.470619 15073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "865b976bf8f845f1b4abd4c2d74a592e"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "196c7e6fd866011b9bc06b76a47b9b97"
server_key_iv: "a795135a1bea65d057f3c4e4a2b5bd2d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.470903 15073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "865b976bf8f845f1b4abd4c2d74a592e"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "196c7e6fd866011b9bc06b76a47b9b97"
server_key_iv: "a795135a1bea65d057f3c4e4a2b5bd2d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.472086 15073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:50.472764 15095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.472899 15073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:50.472988 15073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "865b976bf8f845f1b4abd4c2d74a592e"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "196c7e6fd866011b9bc06b76a47b9b97"
server_key_iv: "a795135a1bea65d057f3c4e4a2b5bd2d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.473052 15073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:50.486109 15073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:50.487627 15073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:50.487773 15073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:50.487974 15073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:50.488236 15073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:50.488267 15073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.488287 15073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:50.488310 15073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.493810 15073 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:45917
I20260808 07:00:50.493873 15297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:45917 every 8 connection(s)
I20260808 07:00:50.494134 15073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:50.502346 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.494769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:37295 (local address 127.21.28.126:43069)
0808 07:00:50.494822 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:50.494824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:50.495007 (+ 183us) server_negotiation.cc:407] Connection header received
0808 07:00:50.495483 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:50.495485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:50.495497 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:50.495549 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:50.496475 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.496989 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.497923 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.498130 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.498185 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.500085 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:50.500086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:50.500087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:50.500093 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:50.501115 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.501463 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.501464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.501465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.501516 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.501718 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.501719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.501720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.501828 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:50.501883 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:50.502124 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 07:00:50.502215 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 07:00:50.502403 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 15073
I20260808 07:00:50.502490 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.494748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:43069 (local address 127.21.28.65:37295)
0808 07:00:50.494895 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:50.494927 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:50.495395 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:50.495612 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:50.495615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:50.496204 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:50.496385 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.496389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.497072 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.497073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.497843 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.497846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.498198 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.498199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.498312 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.498699 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:50.498706 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:50.500002 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:50.501185 (+ 1183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:50.501191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:50.501193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:50.501397 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:50.501571 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:50.501571 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:50.501572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:50.501664 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:50.501882 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:50.501889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:50.502011 (+ 122us) client_negotiation.cc:769] Sending connection context
0808 07:00:50.502095 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 07:00:50.502240 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260808 07:00:50.502509 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 07:00:50.502703 21617 external_mini_cluster.cc:1428] Setting key 33465445f24c2b31b1ea415c8e51b1bd
I20260808 07:00:50.503016 15298 heartbeater.cc:347] Connected to a master server at 127.21.28.126:43069
I20260808 07:00:50.503108 15298 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:50.503268 15298 heartbeater.cc:511] Master 127.21.28.126:43069 requested a full tablet report, sending...
I20260808 07:00:50.503763 14959 ts_manager.cc:194] Registered new tserver with Master: 865b976bf8f845f1b4abd4c2d74a592e (127.21.28.65:45917)
I20260808 07:00:50.504362 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:37295
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 07:00:50.524312 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:43069
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:50.603097 15307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:50.603283 15307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:50.603312 15307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:50.604426 15307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:50.604465 15307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:50.604491 15307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 07:00:50.605886 15307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:50.606093 15307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:50.606290 15307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:50.611001 15319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:50.611039 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:50.611167 15307 server_base.cc:1034] running on GCE node
W20260808 07:00:50.611001 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:50.611416 15307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:50.611634 15307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:50.612787 15307 hybrid_clock.cc:648] HybridClock initialized: now 1786172450612767 us; error 32 us; skew 500 ppm
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:50.615190 15307 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 07:00:50.615605 15307 webserver.cc:460] Webserver started at http://127.21.28.66:40867/ using document root <none> and password file <none>
I20260808 07:00:50.615835 15307 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:50.615883 15307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:50.616008 15307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:50.616966 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "57962b4c91cb4c9ba96ac205f33900fa"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "c6c56e960c94dc1355be55142862fe4e"
server_key_iv: "3da49e28f811c4f5d132d90ff8b69276"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.617281 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "57962b4c91cb4c9ba96ac205f33900fa"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "c6c56e960c94dc1355be55142862fe4e"
server_key_iv: "3da49e28f811c4f5d132d90ff8b69276"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.618477 15307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:00:50.619086 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.619225 15307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:50.619305 15307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "57962b4c91cb4c9ba96ac205f33900fa"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "c6c56e960c94dc1355be55142862fe4e"
server_key_iv: "3da49e28f811c4f5d132d90ff8b69276"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.619367 15307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:50.643750 15307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:50.645531 15307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:50.645665 15307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:50.645879 15307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:50.646175 15307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:50.646207 15307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.646236 15307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:50.646260 15307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.652271 15307 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:33079
I20260808 07:00:50.652325 15514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:33079 every 8 connection(s)
I20260808 07:00:50.652618 15307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:50.658380 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 15307
I20260808 07:00:50.658488 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 07:00:50.658705 21617 external_mini_cluster.cc:1428] Setting key ecef44bc26bef6397f947f3e0248d464
I20260808 07:00:50.661247 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.653275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:53475 (local address 127.21.28.126:43069)
0808 07:00:50.653326 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:50.653328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:50.653538 (+ 210us) server_negotiation.cc:407] Connection header received
0808 07:00:50.654036 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:50.654037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:50.654048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:50.654097 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:50.654967 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.655466 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.656430 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.656613 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.656655 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.658660 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:50.658661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:50.658661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:50.658667 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:50.659921 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.660345 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.660345 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.660345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.660391 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.660618 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.660619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.660619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.660710 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:50.660753 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:50.661065 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 07:00:50.661154 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 07:00:50.661399 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.653236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:43069 (local address 127.21.28.66:53475)
0808 07:00:50.653432 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:50.653466 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:50.653958 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:50.654147 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:50.654150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:50.654702 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:50.654892 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.654897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.655528 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.655528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.656361 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.656363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.656664 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.656665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.656764 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.657160 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:50.657168 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:50.658576 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:50.659992 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:50.659999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:50.660002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:50.660284 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:50.660446 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:50.660447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:50.660448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:50.660569 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:50.660758 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:50.660763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:50.660910 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 07:00:50.661019 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 07:00:50.661192 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":127,"threads_started":1}
I20260808 07:00:50.661859 15517 heartbeater.cc:347] Connected to a master server at 127.21.28.126:43069
I20260808 07:00:50.661962 15517 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:50.662125 15517 heartbeater.cc:511] Master 127.21.28.126:43069 requested a full tablet report, sending...
I20260808 07:00:50.662468 14959 ts_manager.cc:194] Registered new tserver with Master: 57962b4c91cb4c9ba96ac205f33900fa (127.21.28.66:33079)
I20260808 07:00:50.662823 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:53475
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 07:00:50.682250 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:43069
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:50.760512 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:50.760696 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:50.760726 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:50.761804 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:50.761843 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:50.761878 15531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 07:00:50.763260 15531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:50.763481 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:50.763702 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:50.768319 15540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:50.768383 15541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:50.768333 15543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:50.768627 15531 server_base.cc:1034] running on GCE node
I20260808 07:00:50.768788 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:50.768990 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:50.770125 15531 hybrid_clock.cc:648] HybridClock initialized: now 1786172450770112 us; error 26 us; skew 500 ppm
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:50.772363 15531 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 07:00:50.772740 15531 webserver.cc:460] Webserver started at http://127.21.28.67:34907/ using document root <none> and password file <none>
I20260808 07:00:50.772950 15531 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:50.772995 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:50.773115 15531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:50.774048 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "19b0d1c8748542adbfb0879fca9e134d"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "ff68303d7ae17972c65e94fdf9edef99"
server_key_iv: "9b8a0ef10c5cb1ef62c1d5025742345d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.774353 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "19b0d1c8748542adbfb0879fca9e134d"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "ff68303d7ae17972c65e94fdf9edef99"
server_key_iv: "9b8a0ef10c5cb1ef62c1d5025742345d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.775525 15531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:00:50.776235 15555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.776366 15531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.776437 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "19b0d1c8748542adbfb0879fca9e134d"
format_stamp: "Formatted at 2026-08-08 07:00:50 on dist-test-slave-nj21"
server_key: "ff68303d7ae17972c65e94fdf9edef99"
server_key_iv: "9b8a0ef10c5cb1ef62c1d5025742345d"
server_key_version: "encryptionkey@0"
I20260808 07:00:50.776502 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:50.791752 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:50.793381 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:50.793505 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:50.793694 15531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:50.793946 15531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:50.793977 15531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.793995 15531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:50.794018 15531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:50.799714 15531 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:34751
I20260808 07:00:50.799775 15736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:34751 every 8 connection(s)
I20260808 07:00:50.800040 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
Aug 08 07:00:50 dist-test-slave-nj21 krb5kdc[14875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172450, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:50.806164 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 15531
I20260808 07:00:50.806252 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 07:00:50.806460 21617 external_mini_cluster.cc:1428] Setting key d5421a1750cb5358ec74bed7d3c7c5b3
I20260808 07:00:50.808559 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.800640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:42935 (local address 127.21.28.126:43069)
0808 07:00:50.800691 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:50.800693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:50.800905 (+ 212us) server_negotiation.cc:407] Connection header received
0808 07:00:50.801558 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:50.801559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:50.801571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:50.801633 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:50.802464 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.802953 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.803889 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.804059 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.804101 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.806013 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:50.806015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:50.806015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:50.806022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:50.807324 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.807664 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.807665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.807666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.807738 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.807969 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.807970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.807971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.808077 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:50.808131 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:50.808387 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 07:00:50.808471 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 07:00:50.808694 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.800592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:43069 (local address 127.21.28.67:42935)
0808 07:00:50.800782 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:50.800826 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:50.801450 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:50.801696 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:50.801699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:50.802221 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:50.802394 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.802399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.803013 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.803014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.803835 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.803837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.804108 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.804109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.804195 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.804559 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:50.804566 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:50.805918 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:50.807398 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:50.807405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:50.807407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:50.807602 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:50.807793 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:50.807793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:50.807794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:50.807915 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:50.808130 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:50.808136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:50.808274 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 07:00:50.808356 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 07:00:50.808499 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260808 07:00:50.809079 15737 heartbeater.cc:347] Connected to a master server at 127.21.28.126:43069
I20260808 07:00:50.809155 15737 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:50.809284 15737 heartbeater.cc:511] Master 127.21.28.126:43069 requested a full tablet report, sending...
I20260808 07:00:50.809587 14959 ts_manager.cc:194] Registered new tserver with Master: 19b0d1c8748542adbfb0879fca9e134d (127.21.28.67:34751)
I20260808 07:00:50.809939 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:42935
I20260808 07:00:50.818079 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:00:50.824496 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.819434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.21.28.126:43069)
0808 07:00:50.819496 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:50.819497 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:50.819581 (+ 84us) server_negotiation.cc:407] Connection header received
0808 07:00:50.819626 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:50.819627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:50.819634 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:50.819714 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:50.820227 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.820666 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.821304 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.821483 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.821521 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.822471 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:50.822472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:50.822472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:50.822478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:50.823638 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.823964 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.823965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.823965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.824015 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:50.824130 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:50.824130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:50.824131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:50.824245 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:50.824309 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:50.824382 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 07:00:50.824436 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 07:00:50.825819 14959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:00:50.827292 14959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:00:50.832971 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.829145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:45917 (local address 127.0.0.1:35320)
0808 07:00:50.829545 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:50.829554 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:50.829610 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:50.829839 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:50.829840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:50.829842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:50.829881 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.829883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.830776 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.830777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.831821 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.831824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.832692 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.832693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.832770 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.832782 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 07:00:50.832844 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 07:00:50.832898 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260808 07:00:50.832984 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.829347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.21.28.65:45917)
0808 07:00:50.829567 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:50.829571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:50.829618 (+ 47us) server_negotiation.cc:407] Connection header received
0808 07:00:50.829674 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:50.829676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:50.829693 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:50.829768 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:50.829945 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.830697 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.831921 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.832626 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.832682 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.832739 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:50.832864 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 07:00:50.832913 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
I20260808 07:00:50.833236 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.829147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:34751 (local address 127.0.0.1:56912)
0808 07:00:50.829675 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:50.829683 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:50.829721 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:50.830069 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:50.830070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:50.830073 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:50.830115 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.830117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.831322 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.831323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.832194 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.832196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.833012 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.833013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.833076 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.833080 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:50.833128 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 07:00:50.833165 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":29,"thread_start_us":78,"threads_started":1}
I20260808 07:00:50.833483 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.829343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.21.28.67:34751)
0808 07:00:50.829871 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:50.829874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:50.829882 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:00:50.829915 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:50.829917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:50.829931 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:50.829999 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:50.830186 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.831226 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.832262 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.832851 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.832926 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.832977 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:50.833372 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 07:00:50.833413 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":82,"threads_started":1}
I20260808 07:00:50.833639 15179 tablet_service.cc:1467] Processing CreateTablet for tablet 8e07c92e280a41f58350a34c2a958d43 (DEFAULT_TABLE table=test-table [id=7c6933d986664178a031f1dea82e3ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:50.833920 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.829145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:33079 (local address 127.0.0.1:50270)
0808 07:00:50.829290 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:50.829326 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:50.829385 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:50.829771 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:50.829774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:50.829776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:50.829834 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.829835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.831294 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.831295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.832422 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:50.832425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.833695 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.833697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:50.833758 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.833762 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:50.833809 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 07:00:50.833846 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":31,"thread_start_us":53,"threads_started":1}
I20260808 07:00:50.833935 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:50.829338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.21.28.66:33079)
0808 07:00:50.829631 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:50.829634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:50.829641 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:00:50.829671 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:50.829672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:50.829684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:50.829750 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:50.830037 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.831178 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.832503 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:50.833624 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:50.833687 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:50.833738 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:50.833820 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 07:00:50.833860 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260808 07:00:50.834013 15179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e07c92e280a41f58350a34c2a958d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:50.834074 15635 tablet_service.cc:1467] Processing CreateTablet for tablet 8e07c92e280a41f58350a34c2a958d43 (DEFAULT_TABLE table=test-table [id=7c6933d986664178a031f1dea82e3ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:50.834367 15635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e07c92e280a41f58350a34c2a958d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:50.834486 15425 tablet_service.cc:1467] Processing CreateTablet for tablet 8e07c92e280a41f58350a34c2a958d43 (DEFAULT_TABLE table=test-table [id=7c6933d986664178a031f1dea82e3ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:50.834722 15425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e07c92e280a41f58350a34c2a958d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:50.837387 15763 tablet_bootstrap.cc:492] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Bootstrap starting.
I20260808 07:00:50.837466 15762 tablet_bootstrap.cc:492] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e: Bootstrap starting.
I20260808 07:00:50.837754 15761 tablet_bootstrap.cc:492] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Bootstrap starting.
I20260808 07:00:50.837980 15763 tablet_bootstrap.cc:654] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:50.838109 15762 tablet_bootstrap.cc:654] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:50.838388 15761 tablet_bootstrap.cc:654] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:50.838459 15763 log.cc:824] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:50.838658 15762 log.cc:824] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:50.838938 15761 log.cc:824] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:50.839174 15763 tablet_bootstrap.cc:492] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: No bootstrap required, opened a new log
I20260808 07:00:50.839246 15763 ts_tablet_manager.cc:1397] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:50.839321 15762 tablet_bootstrap.cc:492] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e: No bootstrap required, opened a new log
I20260808 07:00:50.839380 15762 ts_tablet_manager.cc:1397] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:50.839564 15761 tablet_bootstrap.cc:492] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: No bootstrap required, opened a new log
I20260808 07:00:50.839620 15761 ts_tablet_manager.cc:1397] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:50.841298 15763 raft_consensus.cc:348] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:50.841234 15761 raft_consensus.cc:348] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:50.841341 15762 raft_consensus.cc:348] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:50.841475 15763 raft_consensus.cc:374] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:50.841485 15762 raft_consensus.cc:374] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:50.841482 15761 raft_consensus.cc:374] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:50.841518 15762 raft_consensus.cc:729] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865b976bf8f845f1b4abd4c2d74a592e, State: Initialized, Role: FOLLOWER
I20260808 07:00:50.841518 15763 raft_consensus.cc:729] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b0d1c8748542adbfb0879fca9e134d, State: Initialized, Role: FOLLOWER
I20260808 07:00:50.841524 15761 raft_consensus.cc:729] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57962b4c91cb4c9ba96ac205f33900fa, State: Initialized, Role: FOLLOWER
I20260808 07:00:50.841643 15761 consensus_queue.cc:260] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:50.841663 15763 consensus_queue.cc:260] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:50.841696 15762 consensus_queue.cc:260] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:50.841887 15761 ts_tablet_manager.cc:1428] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 07:00:50.841939 15763 ts_tablet_manager.cc:1428] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:00:50.841933 15762 ts_tablet_manager.cc:1428] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:50.841998 15517 heartbeater.cc:503] Master 127.21.28.126:43069 was elected leader, sending a full tablet report...
I20260808 07:00:50.842204 15298 heartbeater.cc:503] Master 127.21.28.126:43069 was elected leader, sending a full tablet report...
I20260808 07:00:50.842231 15737 heartbeater.cc:503] Master 127.21.28.126:43069 was elected leader, sending a full tablet report...
W20260808 07:00:50.903182 15520 tablet.cc:2369] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:00:50.994777 15302 tablet.cc:2369] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:51.040791 15768 raft_consensus.cc:482] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:51.040946 15768 raft_consensus.cc:504] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:51.041335 15768 leader_election.cc:288] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865b976bf8f845f1b4abd4c2d74a592e (127.21.28.65:45917), 19b0d1c8748542adbfb0879fca9e134d (127.21.28.67:34751)
I20260808 07:00:51.044909 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.041594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:59599 (local address 127.21.28.67:34751)
0808 07:00:51.041643 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.041645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.041759 (+ 114us) server_negotiation.cc:407] Connection header received
0808 07:00:51.041814 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.041815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.041823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.041870 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:51.042029 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.042731 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.044003 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.044575 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.044630 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.044639 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.044801 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.044838 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 07:00:51.044914 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.041602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:60409 (local address 127.21.28.65:45917)
0808 07:00:51.041643 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.041645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.041657 (+ 12us) server_negotiation.cc:407] Connection header received
0808 07:00:51.041686 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.041687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.041698 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.041744 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:51.041879 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.042635 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.044022 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.044561 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.044630 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.044639 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.044801 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.044838 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 07:00:51.044978 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.041499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:45917 (local address 127.21.28.66:60409)
0808 07:00:51.041575 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.041586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.041634 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.041752 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.041752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.041754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:51.041822 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.041824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.042701 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.042702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.043911 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.043914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.044641 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.044642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.044742 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.044744 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.044791 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.044846 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":2}
I20260808 07:00:51.044983 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.041509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:34751 (local address 127.21.28.66:59599)
0808 07:00:51.041692 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.041701 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.041762 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.041871 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.041871 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.041872 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:51.041945 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.041946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.042799 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.042799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.043911 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.043914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.044642 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.044642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.044736 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.044741 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.044791 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.044845 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":41,"thread_start_us":102,"threads_started":1}
I20260808 07:00:51.045316 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e07c92e280a41f58350a34c2a958d43" candidate_uuid: "57962b4c91cb4c9ba96ac205f33900fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865b976bf8f845f1b4abd4c2d74a592e" is_pre_election: true
I20260808 07:00:51.045359 15664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e07c92e280a41f58350a34c2a958d43" candidate_uuid: "57962b4c91cb4c9ba96ac205f33900fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b0d1c8748542adbfb0879fca9e134d" is_pre_election: true
I20260808 07:00:51.045464 15222 raft_consensus.cc:2454] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57962b4c91cb4c9ba96ac205f33900fa in term 0.
I20260808 07:00:51.045517 15664 raft_consensus.cc:2454] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57962b4c91cb4c9ba96ac205f33900fa in term 0.
I20260808 07:00:51.045672 15343 leader_election.cc:302] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57962b4c91cb4c9ba96ac205f33900fa, 865b976bf8f845f1b4abd4c2d74a592e; no voters:
I20260808 07:00:51.045789 15768 raft_consensus.cc:2781] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:00:51.045845 15768 raft_consensus.cc:482] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:00:51.045876 15768 raft_consensus.cc:3037] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:51.046602 15768 raft_consensus.cc:504] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:51.046711 15768 leader_election.cc:288] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [CANDIDATE]: Term 1 election: Requested vote from peers 865b976bf8f845f1b4abd4c2d74a592e (127.21.28.65:45917), 19b0d1c8748542adbfb0879fca9e134d (127.21.28.67:34751)
I20260808 07:00:51.046838 15664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e07c92e280a41f58350a34c2a958d43" candidate_uuid: "57962b4c91cb4c9ba96ac205f33900fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b0d1c8748542adbfb0879fca9e134d"
I20260808 07:00:51.046864 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e07c92e280a41f58350a34c2a958d43" candidate_uuid: "57962b4c91cb4c9ba96ac205f33900fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865b976bf8f845f1b4abd4c2d74a592e"
I20260808 07:00:51.046917 15664 raft_consensus.cc:3037] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:51.046943 15222 raft_consensus.cc:3037] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:51.047640 15664 raft_consensus.cc:2454] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57962b4c91cb4c9ba96ac205f33900fa in term 1.
I20260808 07:00:51.047640 15222 raft_consensus.cc:2454] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57962b4c91cb4c9ba96ac205f33900fa in term 1.
I20260808 07:00:51.047874 15338 leader_election.cc:302] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b0d1c8748542adbfb0879fca9e134d, 57962b4c91cb4c9ba96ac205f33900fa; no voters:
I20260808 07:00:51.047972 15768 raft_consensus.cc:2781] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:51.048091 15768 raft_consensus.cc:686] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 1 LEADER]: Becoming Leader. State: Replica: 57962b4c91cb4c9ba96ac205f33900fa, State: Running, Role: LEADER
I20260808 07:00:51.048168 15768 consensus_queue.cc:237] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } }
I20260808 07:00:51.048924 14957 catalog_manager.cc:5495] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 57962b4c91cb4c9ba96ac205f33900fa (127.21.28.66). New cstate: current_term: 1 leader_uuid: "57962b4c91cb4c9ba96ac205f33900fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57962b4c91cb4c9ba96ac205f33900fa" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 33079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 } health_report { overall_health: UNKNOWN } } }
W20260808 07:00:51.050489 15739 tablet.cc:2369] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:51.055328 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.053459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.21.28.66:33079)
0808 07:00:51.053500 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.053502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.053511 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:51.053554 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.053555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.053569 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.053627 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:00:51.053720 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.054150 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.054804 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.054977 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.055019 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.055116 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:00:51.055202 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:00:51.055250 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.055263 (+ 13us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.055278 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 07:00:51.056875 15664 raft_consensus.cc:1261] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 1 FOLLOWER]: Refusing update from remote peer 57962b4c91cb4c9ba96ac205f33900fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:51.056876 15222 raft_consensus.cc:1261] T 8e07c92e280a41f58350a34c2a958d43 P 865b976bf8f845f1b4abd4c2d74a592e [term 1 FOLLOWER]: Refusing update from remote peer 57962b4c91cb4c9ba96ac205f33900fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:51.057166 15773 consensus_queue.cc:1035] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b0d1c8748542adbfb0879fca9e134d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:51.057286 15768 consensus_queue.cc:1035] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "865b976bf8f845f1b4abd4c2d74a592e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:51.060338 15784 mvcc.cc:205] Tried to move back new op lower bound from 7316162359526629376 to 7316162359493685248. Current Snapshot: MvccSnapshot[applied={T|T < 7316162359526629376}]
I20260808 07:00:51.061548 14957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:51.061628 14957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:51.062288 14957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f14d19098d34575b5274c296f78ac60: Sending DeleteTablet for 3 replicas of tablet 8e07c92e280a41f58350a34c2a958d43
I20260808 07:00:51.062579 15425 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e07c92e280a41f58350a34c2a958d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:51 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:50270
I20260808 07:00:51.062608 15179 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e07c92e280a41f58350a34c2a958d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:51 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:35320
I20260808 07:00:51.062733 15635 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e07c92e280a41f58350a34c2a958d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:51 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:56912
I20260808 07:00:51.062958 15801 tablet_replica.cc:330] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: stopping tablet replica
I20260808 07:00:51.063145 15801 raft_consensus.cc:2229] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:51.063146 15797 tablet_replica.cc:330] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: stopping tablet replica
I20260808 07:00:51.063237 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 15073
I20260808 07:00:51.063251 15801 raft_consensus.cc:2258] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:51.075378 15801 ts_tablet_manager.cc:1905] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:51.076898 15801 ts_tablet_manager.cc:1918] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:51.076965 15801 log.cc:1196] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/8e07c92e280a41f58350a34c2a958d43
I20260808 07:00:51.077191 15801 ts_tablet_manager.cc:1939] T 8e07c92e280a41f58350a34c2a958d43 P 19b0d1c8748542adbfb0879fca9e134d: Deleting consensus metadata
I20260808 07:00:51.077556 14938 catalog_manager.cc:4841] TS 19b0d1c8748542adbfb0879fca9e134d (127.21.28.67:34751): tablet 8e07c92e280a41f58350a34c2a958d43 (table test-table [id=7c6933d986664178a031f1dea82e3ac2]) successfully deleted
I20260808 07:00:51.095080 15797 raft_consensus.cc:2229] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:00:51.095278 15797 raft_consensus.cc:2258] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:51.096590 15797 ts_tablet_manager.cc:1905] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:51.098011 15797 ts_tablet_manager.cc:1918] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:51.098078 15797 log.cc:1196] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/8e07c92e280a41f58350a34c2a958d43
I20260808 07:00:51.098315 15797 ts_tablet_manager.cc:1939] T 8e07c92e280a41f58350a34c2a958d43 P 57962b4c91cb4c9ba96ac205f33900fa: Deleting consensus metadata
I20260808 07:00:51.098704 14939 catalog_manager.cc:4841] TS 57962b4c91cb4c9ba96ac205f33900fa (127.21.28.66:33079): tablet 8e07c92e280a41f58350a34c2a958d43 (table test-table [id=7c6933d986664178a031f1dea82e3ac2]) successfully deleted
W20260808 07:00:51.100521 14941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:45917 (error 108)
W20260808 07:00:51.100977 15753 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:00:51.100865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:45917 (local address 127.0.0.1:35330)
0808 07:00:51.100912 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.100944 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:45917: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 07:00:51.101083 14941 catalog_manager.cc:4584] TS 865b976bf8f845f1b4abd4c2d74a592e (127.21.28.65:45917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e07c92e280a41f58350a34c2a958d43: Network error: Client connection negotiation failed: client connection to 127.21.28.65:45917: connect: Connection reset by peer (error 104)
I20260808 07:00:51.101481 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 15307
I20260808 07:00:51.113917 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 15531
I20260808 07:00:51.132332 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 14900
2026-08-08T07:00:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2188 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T07:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:00:51Z Disabled control of system clock
I20260808 07:00:51.161868 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:44247
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:44247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:51.240034 15812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:51.240228 15812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:51.240247 15812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:51.241293 15812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:51.241325 15812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:51.241338 15812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:51.241348 15812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:51.241361 15812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:51.242650 15812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:44247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:44247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:51.242827 15812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:51.243125 15812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:51.248270 15829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:51.248275 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:51.248276 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:51.248514 15812 server_base.cc:1034] running on GCE node
I20260808 07:00:51.248741 15812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:51.248997 15812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:51.250133 15812 hybrid_clock.cc:648] HybridClock initialized: now 1786172451250120 us; error 29 us; skew 500 ppm
I20260808 07:00:51.251210 15812 webserver.cc:460] Webserver started at http://127.21.28.126:42053/ using document root <none> and password file <none>
I20260808 07:00:51.251420 15812 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:51.251466 15812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:51.251590 15812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:51.252586 15812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "9580e2836efe45e78e84ae67830b2a23"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "5723a0a92fc388a3b4cd531a026c460a"
server_key_iv: "ac9baabd567ecde64b67f20817ca290c"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.252900 15812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "9580e2836efe45e78e84ae67830b2a23"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "5723a0a92fc388a3b4cd531a026c460a"
server_key_iv: "ac9baabd567ecde64b67f20817ca290c"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.254103 15812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:00:51.254736 15839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.254878 15812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:51.254961 15812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "9580e2836efe45e78e84ae67830b2a23"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "5723a0a92fc388a3b4cd531a026c460a"
server_key_iv: "ac9baabd567ecde64b67f20817ca290c"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.255024 15812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:51.268671 15812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:51.268923 15812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:51.269042 15812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:51.272691 15812 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:44247
I20260808 07:00:51.272720 15931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:44247 every 8 connection(s)
I20260808 07:00:51.273016 15812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:51.273849 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:51.275486 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 15812
I20260808 07:00:51.275585 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:00:51.275810 21617 external_mini_cluster.cc:1428] Setting key 7d098a8305e9a2899ee7793028466c20
I20260808 07:00:51.276180 15932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23: Bootstrap starting.
I20260808 07:00:51.276785 15932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:51.277055 15932 log.cc:824] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:51.277645 15932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23: No bootstrap required, opened a new log
I20260808 07:00:51.279284 15932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9580e2836efe45e78e84ae67830b2a23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44247 } }
I20260808 07:00:51.279402 15932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:51.279434 15932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9580e2836efe45e78e84ae67830b2a23, State: Initialized, Role: FOLLOWER
I20260808 07:00:51.279570 15932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9580e2836efe45e78e84ae67830b2a23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44247 } }
I20260808 07:00:51.279640 15932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:51.279690 15932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:51.279734 15932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:51.280251 15932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9580e2836efe45e78e84ae67830b2a23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44247 } }
I20260808 07:00:51.280346 15932 leader_election.cc:302] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9580e2836efe45e78e84ae67830b2a23; no voters:
I20260808 07:00:51.280503 15932 leader_election.cc:288] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:00:51.280555 15942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:51.280705 15942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [term 1 LEADER]: Becoming Leader. State: Replica: 9580e2836efe45e78e84ae67830b2a23, State: Running, Role: LEADER
I20260808 07:00:51.280757 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.276279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.21.28.126:44247)
0808 07:00:51.276466 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.276473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.276496 (+ 23us) server_negotiation.cc:407] Connection header received
0808 07:00:51.276911 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.276919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.276979 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.277096 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:51.277311 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.278205 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.278944 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.279334 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.279436 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.280069 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:51.280072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:51.280072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:51.280222 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:51.280252 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:51.280262 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:51.280314 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.280418 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.280498 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260808 07:00:51.280767 15932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:51.280812 15942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9580e2836efe45e78e84ae67830b2a23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44247 } }
I20260808 07:00:51.282317 15944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9580e2836efe45e78e84ae67830b2a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9580e2836efe45e78e84ae67830b2a23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44247 } } }
I20260808 07:00:51.282481 15943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9580e2836efe45e78e84ae67830b2a23. Latest consensus state: current_term: 1 leader_uuid: "9580e2836efe45e78e84ae67830b2a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9580e2836efe45e78e84ae67830b2a23" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 44247 } } }
I20260808 07:00:51.282619 15943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:51.282606 15944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23 [sys.catalog]: This master's current role is: LEADER
W20260808 07:00:51.282776 15962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:00:51.282824 15962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:00:51.282877 15974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:51.283038 15974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:51.284329 15974 catalog_manager.cc:1347] Generated new cluster ID: b5dd5a5c35f548dd80f161c93a9f642a
I20260808 07:00:51.284379 15974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:00:51.318467 15974 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:00:51.319164 15974 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:51.322609 15974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23: Generated new TSK 0
I20260808 07:00:51.322770 15974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:00:51.329735 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44247
--builtin_ntp_servers=127.21.28.84:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:51.408528 15976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:51.408718 15976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:51.408748 15976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:51.409870 15976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:51.409909 15976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:51.409941 15976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 07:00:51.411242 15976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:51.411476 15976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:51.411767 15976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:51.416412 15983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:51.416436 15982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:51.416476 15976 server_base.cc:1034] running on GCE node
W20260808 07:00:51.416440 15987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:51.416854 15976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:51.417059 15976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:51.418195 15976 hybrid_clock.cc:648] HybridClock initialized: now 1786172451418184 us; error 29 us; skew 500 ppm
I20260808 07:00:51.419306 15976 webserver.cc:460] Webserver started at http://127.21.28.65:36141/ using document root <none> and password file <none>
I20260808 07:00:51.419521 15976 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:51.419564 15976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:51.419715 15976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:51.420704 15976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "c33ce518dbd04b9f9def7963b55a301b"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "3e9eb9dcd68f410db2ca79e062b11640"
server_key_iv: "04896ae506c8a2b3d2cdc954d51ac31c"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.421017 15976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "c33ce518dbd04b9f9def7963b55a301b"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "3e9eb9dcd68f410db2ca79e062b11640"
server_key_iv: "04896ae506c8a2b3d2cdc954d51ac31c"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.422144 15976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:00:51.422729 15998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.422881 15976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:51.422971 15976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "c33ce518dbd04b9f9def7963b55a301b"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "3e9eb9dcd68f410db2ca79e062b11640"
server_key_iv: "04896ae506c8a2b3d2cdc954d51ac31c"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.423022 15976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:51.441727 15976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:51.441951 15976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:51.442040 15976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:51.442198 15976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:51.442440 15976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:51.442466 15976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.442485 15976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:51.442502 15976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.447715 15976 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:32851
I20260808 07:00:51.447779 16189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:32851 every 8 connection(s)
I20260808 07:00:51.448041 15976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 07:00:51.452430 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.448653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:60291 (local address 127.21.28.126:44247)
0808 07:00:51.448726 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.448728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.448879 (+ 151us) server_negotiation.cc:407] Connection header received
0808 07:00:51.449307 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.449308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.449316 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.449359 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:51.449948 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.450427 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.451326 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.451537 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.451585 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.452239 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:51.452240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:51.452240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:51.452244 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:51.452257 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:51.452261 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:51.452299 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.452347 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.452368 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 07:00:51.452608 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.448629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:44247 (local address 127.21.28.65:60291)
0808 07:00:51.448779 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.448810 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.449239 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.449422 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.449425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.449432 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:51.449890 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.449893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.450483 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.450483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.451268 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.451270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.451592 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.451593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.451720 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.452129 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:51.452129 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:51.452154 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:51.452174 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:51.452299 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:51.452301 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:51.452303 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.452343 (+ 40us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.452420 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260808 07:00:51.453053 16192 heartbeater.cc:347] Connected to a master server at 127.21.28.126:44247
I20260808 07:00:51.453126 16192 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:51.453256 16192 heartbeater.cc:511] Master 127.21.28.126:44247 requested a full tablet report, sending...
I20260808 07:00:51.453684 15869 ts_manager.cc:194] Registered new tserver with Master: c33ce518dbd04b9f9def7963b55a301b (127.21.28.65:32851)
I20260808 07:00:51.454276 15869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.65:60291
I20260808 07:00:51.454901 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 15976
I20260808 07:00:51.454999 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 07:00:51.455152 21617 external_mini_cluster.cc:1428] Setting key 14b493f6fca56b2798e053ca489b3c6a
I20260808 07:00:51.455859 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44247
--builtin_ntp_servers=127.21.28.84:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:51.534965 16197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:51.535147 16197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:51.535166 16197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:51.536242 16197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:51.536275 16197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:51.536298 16197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 07:00:51.537557 16197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:51.537729 16197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:51.537966 16197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:51.542513 16207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:51.542620 16197 server_base.cc:1034] running on GCE node
W20260808 07:00:51.542527 16205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:51.542536 16204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:51.542929 16197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:51.543177 16197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:51.544306 16197 hybrid_clock.cc:648] HybridClock initialized: now 1786172451544281 us; error 38 us; skew 500 ppm
I20260808 07:00:51.545498 16197 webserver.cc:460] Webserver started at http://127.21.28.66:33585/ using document root <none> and password file <none>
I20260808 07:00:51.545709 16197 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:51.545753 16197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:51.545872 16197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:51.546823 16197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "5293e91d958c4db5b08acf12780668be"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "d777f72b2115c789d363b3f2b78f0507"
server_key_iv: "13b44cead05745094f406b3a1651ee7b"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.547123 16197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "5293e91d958c4db5b08acf12780668be"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "d777f72b2115c789d363b3f2b78f0507"
server_key_iv: "13b44cead05745094f406b3a1651ee7b"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.548345 16197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:00:51.548967 16217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.549125 16197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:51.549209 16197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "5293e91d958c4db5b08acf12780668be"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "d777f72b2115c789d363b3f2b78f0507"
server_key_iv: "13b44cead05745094f406b3a1651ee7b"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.549257 16197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:51.565295 16197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:51.565528 16197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:51.565620 16197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:51.565788 16197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:51.566045 16197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:51.566073 16197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.566097 16197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:51.566113 16197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.571655 16197 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:37421
I20260808 07:00:51.571720 16408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:37421 every 8 connection(s)
I20260808 07:00:51.572014 16197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 07:00:51.576793 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.572655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:37731 (local address 127.21.28.126:44247)
0808 07:00:51.572707 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.572709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.572928 (+ 219us) server_negotiation.cc:407] Connection header received
0808 07:00:51.573407 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.573409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.573422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.573470 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:51.574103 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.574588 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.575513 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.575741 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.575799 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.576543 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:51.576545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:51.576545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:51.576550 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:51.576566 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:51.576570 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:51.576615 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.576688 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.576713 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 07:00:51.576975 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.572638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:44247 (local address 127.21.28.66:37731)
0808 07:00:51.572816 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.572847 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.573315 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.573543 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.573546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.573554 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:51.574019 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.574023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.574667 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.574668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.575433 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.575435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.575811 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.575812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.575933 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.576396 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:51.576396 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:51.576443 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:51.576464 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:51.576620 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:51.576625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:51.576628 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.576686 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.576783 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260808 07:00:51.577399 16409 heartbeater.cc:347] Connected to a master server at 127.21.28.126:44247
I20260808 07:00:51.577481 16409 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:51.577646 16409 heartbeater.cc:511] Master 127.21.28.126:44247 requested a full tablet report, sending...
I20260808 07:00:51.577982 15869 ts_manager.cc:194] Registered new tserver with Master: 5293e91d958c4db5b08acf12780668be (127.21.28.66:37421)
I20260808 07:00:51.578321 15869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.66:37731
I20260808 07:00:51.580785 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 16197
I20260808 07:00:51.580873 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 07:00:51.581075 21617 external_mini_cluster.cc:1428] Setting key fd5ddd010b3feda3f94999d89da52f2d
I20260808 07:00:51.581883 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44247
--builtin_ntp_servers=127.21.28.84:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:51.661248 16414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:51.661435 16414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:51.661455 16414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:51.662532 16414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:51.662568 16414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:51.662595 16414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 07:00:51.663911 16414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:51.664106 16414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:51.664355 16414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:51.668895 16432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:51.668866 16430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:51.668877 16434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:51.669020 16414 server_base.cc:1034] running on GCE node
I20260808 07:00:51.669301 16414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:51.669525 16414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:51.670652 16414 hybrid_clock.cc:648] HybridClock initialized: now 1786172451670646 us; error 27 us; skew 500 ppm
I20260808 07:00:51.671887 16414 webserver.cc:460] Webserver started at http://127.21.28.67:39623/ using document root <none> and password file <none>
I20260808 07:00:51.672122 16414 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:51.672169 16414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:51.672297 16414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:51.673254 16414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "2df2bcd71afb42fb9f79d84261f621bf"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "c0fe63445b48323ed93aca8c1b07911d"
server_key_iv: "dcf7438989e6c7a9c686eb0729e918da"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.673576 16414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "2df2bcd71afb42fb9f79d84261f621bf"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "c0fe63445b48323ed93aca8c1b07911d"
server_key_iv: "dcf7438989e6c7a9c686eb0729e918da"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.674712 16414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:00:51.675431 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.675604 16414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:51.675702 16414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "2df2bcd71afb42fb9f79d84261f621bf"
format_stamp: "Formatted at 2026-08-08 07:00:51 on dist-test-slave-nj21"
server_key: "c0fe63445b48323ed93aca8c1b07911d"
server_key_iv: "dcf7438989e6c7a9c686eb0729e918da"
server_key_version: "encryptionkey@0"
I20260808 07:00:51.675769 16414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:51.693077 16414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:51.693315 16414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:51.693432 16414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:51.693630 16414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:51.693921 16414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:51.693953 16414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.693982 16414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:51.694005 16414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:51.699618 16414 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:37983
I20260808 07:00:51.699695 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:37983 every 8 connection(s)
I20260808 07:00:51.699977 16414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 07:00:51.704787 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.700624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:35057 (local address 127.21.28.126:44247)
0808 07:00:51.700681 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.700683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.700877 (+ 194us) server_negotiation.cc:407] Connection header received
0808 07:00:51.701360 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.701362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.701374 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.701443 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:51.702082 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.702579 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.703511 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.703745 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.703802 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.704532 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:51.704533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:51.704534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:51.704538 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:51.704554 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:51.704558 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:51.704602 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.704678 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.704706 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 07:00:51.704969 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.700599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:44247 (local address 127.21.28.67:35057)
0808 07:00:51.700759 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.700791 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.701260 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.701508 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.701512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.701519 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:51.701993 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.701997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.702657 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.702658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.703427 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.703430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.703811 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.703812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.703938 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.704402 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:51.704403 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:51.704432 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:51.704454 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:51.704609 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:51.704614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:51.704617 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.704676 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.704775 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260808 07:00:51.705374 16630 heartbeater.cc:347] Connected to a master server at 127.21.28.126:44247
I20260808 07:00:51.705457 16630 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:51.705627 16630 heartbeater.cc:511] Master 127.21.28.126:44247 requested a full tablet report, sending...
I20260808 07:00:51.705958 15869 ts_manager.cc:194] Registered new tserver with Master: 2df2bcd71afb42fb9f79d84261f621bf (127.21.28.67:37983)
I20260808 07:00:51.706308 15869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.67:35057
I20260808 07:00:51.706845 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 16414
I20260808 07:00:51.706941 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 07:00:51.707120 21617 external_mini_cluster.cc:1428] Setting key ead4496e71621814f310e0a6312dbb37
I20260808 07:00:51.707837 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:00:51.711513 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.709154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.21.28.126:44247)
0808 07:00:51.709190 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.709192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.709284 (+ 92us) server_negotiation.cc:407] Connection header received
0808 07:00:51.709344 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.709344 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.709351 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.709384 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:51.709530 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.709972 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.710637 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.710802 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.710843 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.711364 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:51.711365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:51.711365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:51.711368 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:51.711377 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:51.711380 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:51.711413 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.711446 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.711461 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:00:51.712915 15869 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:00:51.714402 15869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:00:51.720044 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.716286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:32851 (local address 127.0.0.1:33478)
0808 07:00:51.716393 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.716428 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.716489 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.716802 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.716805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.716808 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:51.716872 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.716875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.717902 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.717903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.718982 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.718984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.719749 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.719750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.719829 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.719842 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.719919 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.719973 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
I20260808 07:00:51.720055 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.716464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.21.28.65:32851)
0808 07:00:51.716630 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.716634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.716645 (+ 11us) server_negotiation.cc:407] Connection header received
0808 07:00:51.716683 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.716686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.716701 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.716956 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:51.717011 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.717830 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.719081 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.719652 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.719746 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.719837 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.719935 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.719973 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260808 07:00:51.720530 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.716311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:37421 (local address 127.0.0.1:45016)
0808 07:00:51.716805 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.716813 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.716857 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.717021 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.717022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.717024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:51.717061 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.717062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.718305 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.718306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.719315 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.719317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.720305 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.720306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.720364 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.720368 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.720422 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.720460 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":67,"threads_started":1}
I20260808 07:00:51.720752 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.716550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.21.28.66:37421)
0808 07:00:51.716789 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.716792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.716853 (+ 61us) server_negotiation.cc:407] Connection header received
0808 07:00:51.716927 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.716929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.716944 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.717017 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:51.717114 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.718224 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.719388 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.720132 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.720197 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.720273 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.720634 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.720675 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":129,"threads_started":1}
I20260808 07:00:51.720920 16074 tablet_service.cc:1467] Processing CreateTablet for tablet f3fb4e99bf344cacb1ff1454297047d2 (DEFAULT_TABLE table=test-table [id=64670f7fdf6c403b91fbed070cf3d0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:51.721323 16074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fb4e99bf344cacb1ff1454297047d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:51.721299 16303 tablet_service.cc:1467] Processing CreateTablet for tablet f3fb4e99bf344cacb1ff1454297047d2 (DEFAULT_TABLE table=test-table [id=64670f7fdf6c403b91fbed070cf3d0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:51.721642 16303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fb4e99bf344cacb1ff1454297047d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:51.721827 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.716291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:37983 (local address 127.0.0.1:38812)
0808 07:00:51.716663 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.716672 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.716726 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.716958 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.716959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.716962 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:51.717009 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.717011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.719393 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.719393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.720450 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.720452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.721603 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.721604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.721664 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.721668 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.721722 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.721763 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":82,"threads_started":1}
I20260808 07:00:51.722111 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.716482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.21.28.67:37983)
0808 07:00:51.716665 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.716669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.716716 (+ 47us) server_negotiation.cc:407] Connection header received
0808 07:00:51.716788 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.716790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.716804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.716880 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:51.717079 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.718224 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.720526 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.721528 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.721594 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.721670 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.721997 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.722039 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260808 07:00:51.722740 16522 tablet_service.cc:1467] Processing CreateTablet for tablet f3fb4e99bf344cacb1ff1454297047d2 (DEFAULT_TABLE table=test-table [id=64670f7fdf6c403b91fbed070cf3d0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:51.723090 16522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3fb4e99bf344cacb1ff1454297047d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:51.724305 16652 tablet_bootstrap.cc:492] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Bootstrap starting.
I20260808 07:00:51.724707 16654 tablet_bootstrap.cc:492] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b: Bootstrap starting.
I20260808 07:00:51.724944 16652 tablet_bootstrap.cc:654] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:51.725214 16652 log.cc:824] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:51.725312 16654 tablet_bootstrap.cc:654] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:51.725620 16654 log.cc:824] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:51.725848 16652 tablet_bootstrap.cc:492] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: No bootstrap required, opened a new log
I20260808 07:00:51.725899 16652 ts_tablet_manager.cc:1397] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:51.726200 16657 tablet_bootstrap.cc:492] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: Bootstrap starting.
I20260808 07:00:51.726397 16654 tablet_bootstrap.cc:492] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b: No bootstrap required, opened a new log
I20260808 07:00:51.726477 16654 ts_tablet_manager.cc:1397] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:51.726763 16657 tablet_bootstrap.cc:654] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:51.727032 16657 log.cc:824] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:51.727766 16657 tablet_bootstrap.cc:492] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: No bootstrap required, opened a new log
I20260808 07:00:51.727823 16657 ts_tablet_manager.cc:1397] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:51.728386 16652 raft_consensus.cc:348] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.728549 16652 raft_consensus.cc:374] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:51.728583 16652 raft_consensus.cc:729] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5293e91d958c4db5b08acf12780668be, State: Initialized, Role: FOLLOWER
I20260808 07:00:51.728724 16652 consensus_queue.cc:260] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.728988 16652 ts_tablet_manager.cc:1428] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:51.728899 16654 raft_consensus.cc:348] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.729018 16409 heartbeater.cc:503] Master 127.21.28.126:44247 was elected leader, sending a full tablet report...
I20260808 07:00:51.729032 16654 raft_consensus.cc:374] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:51.729071 16654 raft_consensus.cc:729] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33ce518dbd04b9f9def7963b55a301b, State: Initialized, Role: FOLLOWER
I20260808 07:00:51.729175 16654 consensus_queue.cc:260] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.729454 16654 ts_tablet_manager.cc:1428] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:51.729501 16192 heartbeater.cc:503] Master 127.21.28.126:44247 was elected leader, sending a full tablet report...
I20260808 07:00:51.730571 16657 raft_consensus.cc:348] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.730716 16657 raft_consensus.cc:374] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:51.730762 16657 raft_consensus.cc:729] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df2bcd71afb42fb9f79d84261f621bf, State: Initialized, Role: FOLLOWER
I20260808 07:00:51.730865 16657 consensus_queue.cc:260] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.731088 16657 ts_tablet_manager.cc:1428] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:51.731149 16630 heartbeater.cc:503] Master 127.21.28.126:44247 was elected leader, sending a full tablet report...
W20260808 07:00:51.822540 16410 tablet.cc:2369] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:51.834349 16663 raft_consensus.cc:482] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:51.834475 16663 raft_consensus.cc:504] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.834828 16663 leader_election.cc:288] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c33ce518dbd04b9f9def7963b55a301b (127.21.28.65:32851), 5293e91d958c4db5b08acf12780668be (127.21.28.66:37421)
I20260808 07:00:51.837486 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.835093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:54137 (local address 127.21.28.66:37421)
0808 07:00:51.835134 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.835136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.835216 (+ 80us) server_negotiation.cc:407] Connection header received
0808 07:00:51.835271 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.835272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.835282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.835338 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:51.835478 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.835986 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.836853 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.837196 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.837261 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.837269 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.837388 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.837422 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 07:00:51.837505 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.834981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:32851 (local address 127.21.28.67:54635)
0808 07:00:51.835038 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.835051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.835099 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.835245 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.835246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.835248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:51.835323 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.835325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.835971 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.835972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.836733 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.836735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.837244 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.837245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.837316 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.837320 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.837361 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.837401 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 07:00:51.837493 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.835084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:54635 (local address 127.21.28.65:32851)
0808 07:00:51.835131 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.835134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.835143 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:51.835171 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.835172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.835183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.835236 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:51.835392 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.835906 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.836811 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.837171 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.837222 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.837230 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.837375 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.837423 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 07:00:51.837529 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.834981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:37421 (local address 127.21.28.67:54137)
0808 07:00:51.835156 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.835165 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:51.835221 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:51.835346 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:51.835346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:51.835348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:51.835420 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.835421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.836059 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.836060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.836771 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:51.836773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.837271 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.837272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:51.837327 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.837330 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 07:00:51.837377 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 07:00:51.837424 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":12,"thread_start_us":91,"threads_started":1}
I20260808 07:00:51.837785 16114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3fb4e99bf344cacb1ff1454297047d2" candidate_uuid: "2df2bcd71afb42fb9f79d84261f621bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33ce518dbd04b9f9def7963b55a301b" is_pre_election: true
I20260808 07:00:51.837940 16114 raft_consensus.cc:2454] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df2bcd71afb42fb9f79d84261f621bf in term 0.
I20260808 07:00:51.837967 16334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3fb4e99bf344cacb1ff1454297047d2" candidate_uuid: "2df2bcd71afb42fb9f79d84261f621bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5293e91d958c4db5b08acf12780668be" is_pre_election: true
I20260808 07:00:51.838093 16334 raft_consensus.cc:2454] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df2bcd71afb42fb9f79d84261f621bf in term 0.
I20260808 07:00:51.838178 16447 leader_election.cc:302] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df2bcd71afb42fb9f79d84261f621bf, c33ce518dbd04b9f9def7963b55a301b; no voters:
I20260808 07:00:51.838296 16663 raft_consensus.cc:2781] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:00:51.838351 16663 raft_consensus.cc:482] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:00:51.838366 16663 raft_consensus.cc:3037] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:51.839027 16663 raft_consensus.cc:504] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.839130 16663 leader_election.cc:288] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [CANDIDATE]: Term 1 election: Requested vote from peers c33ce518dbd04b9f9def7963b55a301b (127.21.28.65:32851), 5293e91d958c4db5b08acf12780668be (127.21.28.66:37421)
I20260808 07:00:51.839283 16334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3fb4e99bf344cacb1ff1454297047d2" candidate_uuid: "2df2bcd71afb42fb9f79d84261f621bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5293e91d958c4db5b08acf12780668be"
I20260808 07:00:51.839286 16114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3fb4e99bf344cacb1ff1454297047d2" candidate_uuid: "2df2bcd71afb42fb9f79d84261f621bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33ce518dbd04b9f9def7963b55a301b"
I20260808 07:00:51.839354 16114 raft_consensus.cc:3037] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:51.839356 16334 raft_consensus.cc:3037] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:51.840096 16334 raft_consensus.cc:2454] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df2bcd71afb42fb9f79d84261f621bf in term 1.
I20260808 07:00:51.840116 16114 raft_consensus.cc:2454] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df2bcd71afb42fb9f79d84261f621bf in term 1.
I20260808 07:00:51.840260 16447 leader_election.cc:302] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df2bcd71afb42fb9f79d84261f621bf, c33ce518dbd04b9f9def7963b55a301b; no voters:
I20260808 07:00:51.840358 16663 raft_consensus.cc:2781] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:51.840492 16663 raft_consensus.cc:686] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 1 LEADER]: Becoming Leader. State: Replica: 2df2bcd71afb42fb9f79d84261f621bf, State: Running, Role: LEADER
I20260808 07:00:51.840574 16663 consensus_queue.cc:237] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } }
I20260808 07:00:51.841315 15869 catalog_manager.cc:5495] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df2bcd71afb42fb9f79d84261f621bf (127.21.28.67). New cstate: current_term: 1 leader_uuid: "2df2bcd71afb42fb9f79d84261f621bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df2bcd71afb42fb9f79d84261f621bf" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 37983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 } health_report { overall_health: UNKNOWN } } }
I20260808 07:00:51.865033 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.863119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.21.28.67:37983)
0808 07:00:51.863176 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:51.863178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:51.863187 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:51.863222 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:51.863223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:51.863237 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:51.863292 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:00:51.863385 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.863857 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.864513 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:51.864681 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:51.864721 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:51.864808 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:00:51.864892 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:00:51.864944 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:51.864955 (+ 11us) server_negotiation.cc:299] Negotiation successful
0808 07:00:51.864968 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 07:00:51.866567 16114 raft_consensus.cc:1261] T f3fb4e99bf344cacb1ff1454297047d2 P c33ce518dbd04b9f9def7963b55a301b [term 1 FOLLOWER]: Refusing update from remote peer 2df2bcd71afb42fb9f79d84261f621bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:51.866566 16334 raft_consensus.cc:1261] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 1 FOLLOWER]: Refusing update from remote peer 2df2bcd71afb42fb9f79d84261f621bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:51.866860 16663 consensus_queue.cc:1035] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5293e91d958c4db5b08acf12780668be" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:51.867000 16665 consensus_queue.cc:1035] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33ce518dbd04b9f9def7963b55a301b" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 32851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:51.870364 16674 mvcc.cc:205] Tried to move back new op lower bound from 7316162362843279360 to 7316162362739625984. Current Snapshot: MvccSnapshot[applied={T|T < 7316162362843279360}]
I20260808 07:00:51.871479 15868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:51.871570 15868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:51.872277 15868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9580e2836efe45e78e84ae67830b2a23: Sending DeleteTablet for 3 replicas of tablet f3fb4e99bf344cacb1ff1454297047d2
I20260808 07:00:51.872588 16074 tablet_service.cc:1514] Processing DeleteTablet for tablet f3fb4e99bf344cacb1ff1454297047d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:51 UTC) from {username='slave'} at 127.0.0.1:33478
I20260808 07:00:51.872612 16522 tablet_service.cc:1514] Processing DeleteTablet for tablet f3fb4e99bf344cacb1ff1454297047d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:51 UTC) from {username='slave'} at 127.0.0.1:38812
I20260808 07:00:51.872865 16303 tablet_service.cc:1514] Processing DeleteTablet for tablet f3fb4e99bf344cacb1ff1454297047d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:51 UTC) from {username='slave'} at 127.0.0.1:45016
I20260808 07:00:51.872973 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 15976
I20260808 07:00:51.873519 16694 tablet_replica.cc:330] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: stopping tablet replica
I20260808 07:00:51.873633 16694 raft_consensus.cc:2229] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:51.873760 16694 raft_consensus.cc:2258] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:51.879016 16694 ts_tablet_manager.cc:1905] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:51.880615 16694 ts_tablet_manager.cc:1918] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:51.880726 16694 log.cc:1196] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/f3fb4e99bf344cacb1ff1454297047d2
I20260808 07:00:51.881027 16694 ts_tablet_manager.cc:1939] T f3fb4e99bf344cacb1ff1454297047d2 P 5293e91d958c4db5b08acf12780668be: Deleting consensus metadata
I20260808 07:00:51.881435 15846 catalog_manager.cc:4841] TS 5293e91d958c4db5b08acf12780668be (127.21.28.66:37421): tablet f3fb4e99bf344cacb1ff1454297047d2 (table test-table [id=64670f7fdf6c403b91fbed070cf3d0a5]) successfully deleted
I20260808 07:00:51.884970 16692 tablet_replica.cc:330] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: stopping tablet replica
I20260808 07:00:51.885316 16692 raft_consensus.cc:2229] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:00:51.885484 16692 raft_consensus.cc:2258] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:51.885952 16692 ts_tablet_manager.cc:1905] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:51.887507 16692 ts_tablet_manager.cc:1918] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:51.887622 16692 log.cc:1196] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/f3fb4e99bf344cacb1ff1454297047d2
I20260808 07:00:51.887926 16692 ts_tablet_manager.cc:1939] T f3fb4e99bf344cacb1ff1454297047d2 P 2df2bcd71afb42fb9f79d84261f621bf: Deleting consensus metadata
I20260808 07:00:51.888319 15843 catalog_manager.cc:4841] TS 2df2bcd71afb42fb9f79d84261f621bf (127.21.28.67:37983): tablet f3fb4e99bf344cacb1ff1454297047d2 (table test-table [id=64670f7fdf6c403b91fbed070cf3d0a5]) successfully deleted
W20260808 07:00:51.889283 15840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:32851 (error 108)
I20260808 07:00:51.889753 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:51.889640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:32851 (local address 127.0.0.1:33494)
0808 07:00:51.889689 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:51.889725 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 07:00:51.889747 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 16197
W20260808 07:00:51.907235 15840 catalog_manager.cc:4584] TS c33ce518dbd04b9f9def7963b55a301b (127.21.28.65:32851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3fb4e99bf344cacb1ff1454297047d2: Network error: Client connection negotiation failed: client connection to 127.21.28.65:32851: connect: Connection refused (error 111)
I20260808 07:00:51.911435 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 16414
I20260808 07:00:51.922883 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 15812
2026-08-08T07:00:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (792 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:00:51 dist-test-slave-nj21 krb5kdc[16698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:00:51 dist-test-slave-nj21 krb5kdc[16698](info): set up 2 sockets
Aug 08 07:00:51 dist-test-slave-nj21 krb5kdc[16698](info): commencing operation
krb5kdc: starting...
W20260808 07:00:53.053071 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:00:53.162009 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:45115
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:45115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:53.240121 16717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:53.240307 16717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:53.240327 16717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:53.241377 16717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:53.241417 16717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:53.241430 16717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:53.241441 16717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:53.241457 16717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:53.242825 16717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:45115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:45115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:53.243003 16717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:53.243175 16717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:53.247639 16725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:53.247623 16726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:53.247622 16729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:53.247808 16717 server_base.cc:1034] running on GCE node
I20260808 07:00:53.248088 16717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:53.248385 16717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:53.249549 16717 hybrid_clock.cc:648] HybridClock initialized: now 1786172453249513 us; error 27 us; skew 500 ppm
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:53.252243 16717 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:00:53.252630 16717 webserver.cc:460] Webserver started at http://127.21.28.126:41283/ using document root <none> and password file <none>
I20260808 07:00:53.252817 16717 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:53.252857 16717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:53.252985 16717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:53.253952 16717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "f0a291e314e6474b9e702f6412b98e96"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "e922b76b1aae072106258991a948022c"
server_key_iv: "453c82cdf770d6b087af8bb240cfdf09"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.254276 16717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "f0a291e314e6474b9e702f6412b98e96"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "e922b76b1aae072106258991a948022c"
server_key_iv: "453c82cdf770d6b087af8bb240cfdf09"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.255470 16717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:53.256135 16737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.256266 16717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.256350 16717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "f0a291e314e6474b9e702f6412b98e96"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "e922b76b1aae072106258991a948022c"
server_key_iv: "453c82cdf770d6b087af8bb240cfdf09"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.256415 16717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:53.263062 16717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:53.264989 16717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:53.265108 16717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:53.268750 16717 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:45115
I20260808 07:00:53.268780 16832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:45115 every 8 connection(s)
I20260808 07:00:53.269066 16717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:53.269837 16834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:53.271973 16834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96: Bootstrap starting.
I20260808 07:00:53.272480 16834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:53.272701 16834 log.cc:824] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:53.273305 16834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96: No bootstrap required, opened a new log
I20260808 07:00:53.274757 16834 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a291e314e6474b9e702f6412b98e96" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45115 } }
I20260808 07:00:53.274879 16834 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:53.274912 16834 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a291e314e6474b9e702f6412b98e96, State: Initialized, Role: FOLLOWER
I20260808 07:00:53.275017 16834 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a291e314e6474b9e702f6412b98e96" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45115 } }
I20260808 07:00:53.275086 16834 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:53.275115 16834 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:53.275146 16834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:53.275591 16834 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a291e314e6474b9e702f6412b98e96" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45115 } }
I20260808 07:00:53.275696 16834 leader_election.cc:302] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0a291e314e6474b9e702f6412b98e96; no voters:
I20260808 07:00:53.275857 16834 leader_election.cc:288] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:00:53.275861 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 16717
I20260808 07:00:53.275897 16837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:53.275957 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:00:53.276046 16837 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [term 1 LEADER]: Becoming Leader. State: Replica: f0a291e314e6474b9e702f6412b98e96, State: Running, Role: LEADER
I20260808 07:00:53.276108 16834 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:53.276134 16837 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a291e314e6474b9e702f6412b98e96" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45115 } }
I20260808 07:00:53.276188 21617 external_mini_cluster.cc:1428] Setting key c3089d4130842d0b2c0fa3bb83622806
I20260808 07:00:53.276749 16840 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0a291e314e6474b9e702f6412b98e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a291e314e6474b9e702f6412b98e96" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45115 } } }
I20260808 07:00:53.276832 16839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0a291e314e6474b9e702f6412b98e96. Latest consensus state: current_term: 1 leader_uuid: "f0a291e314e6474b9e702f6412b98e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a291e314e6474b9e702f6412b98e96" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 45115 } } }
I20260808 07:00:53.276926 16839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:53.276893 16840 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96 [sys.catalog]: This master's current role is: LEADER
W20260808 07:00:53.277823 16866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:00:53.277876 16866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:00:53.277925 16855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:53.278105 16855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:53.279563 16855 catalog_manager.cc:1347] Generated new cluster ID: 22cb7235170f4e0fb2339359b7c44094
I20260808 07:00:53.279616 16855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:53.284657 16855 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:00:53.285300 16855 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:53.286988 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.276618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.21.28.126:45115)
0808 07:00:53.277339 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:53.277348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:53.277379 (+ 31us) server_negotiation.cc:407] Connection header received
0808 07:00:53.277574 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:53.277581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:53.277640 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:53.277778 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:53.278552 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.279796 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.280783 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.281158 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.281234 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.283642 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:53.283646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:53.283646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:53.283657 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:53.285567 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.285931 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.285933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.285934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.286026 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.286208 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.286209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.286209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.286440 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:53.286570 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:53.286672 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 07:00:53.286797 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":73,"threads_started":1}
I20260808 07:00:53.289484 16855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96: Generated new TSK 0
I20260808 07:00:53.289635 16855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 07:00:53.318527 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45115
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:53.396889 16877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:53.397078 16877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:53.397097 16877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:53.398239 16877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:53.398279 16877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:53.398304 16877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 07:00:53.399751 16877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:53.399971 16877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:53.400179 16877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:53.404851 16888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:53.404863 16885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:53.404973 16877 server_base.cc:1034] running on GCE node
W20260808 07:00:53.404878 16890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:53.405262 16877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:53.405486 16877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:53.406631 16877 hybrid_clock.cc:648] HybridClock initialized: now 1786172453406617 us; error 25 us; skew 500 ppm
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:53.408991 16877 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 07:00:53.409368 16877 webserver.cc:460] Webserver started at http://127.21.28.65:40543/ using document root <none> and password file <none>
I20260808 07:00:53.409591 16877 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:53.409639 16877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:53.409760 16877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:53.410719 16877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "5a5315c877de4e7e9b06ae955f0b481e"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "048a6d336c05298026f3460470119e69"
server_key_iv: "7c5a015763951fd149a8daefb47c757e"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.411037 16877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "5a5315c877de4e7e9b06ae955f0b481e"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "048a6d336c05298026f3460470119e69"
server_key_iv: "7c5a015763951fd149a8daefb47c757e"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.412273 16877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:00:53.412949 16898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.413089 16877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:53.413175 16877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "5a5315c877de4e7e9b06ae955f0b481e"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "048a6d336c05298026f3460470119e69"
server_key_iv: "7c5a015763951fd149a8daefb47c757e"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.413230 16877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:53.430413 16877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:53.432489 16877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:53.432619 16877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:53.432811 16877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:53.433085 16877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:53.433117 16877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.433145 16877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:53.433173 16877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.438699 16877 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:36157
I20260808 07:00:53.438760 17089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:36157 every 8 connection(s)
I20260808 07:00:53.439019 16877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 07:00:53.442787 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 16877
I20260808 07:00:53.442902 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 07:00:53.443102 21617 external_mini_cluster.cc:1428] Setting key 2ea04719462f03aa0cd96c2e5a3bb443
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:53.448273 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.439803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:46719 (local address 127.21.28.126:45115)
0808 07:00:53.439853 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:53.439855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:53.440023 (+ 168us) server_negotiation.cc:407] Connection header received
0808 07:00:53.440514 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:53.440515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:53.440527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:53.440579 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:53.441462 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.442000 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.442982 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.443306 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.443382 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.445623 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:53.445625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:53.445625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:53.445633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:53.446995 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.447345 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.447347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.447347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.447420 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.447611 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.447611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.447612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.447773 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:53.447852 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:53.448071 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 07:00:53.448160 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 07:00:53.448392 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.439720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45115 (local address 127.21.28.65:46719)
0808 07:00:53.439906 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:53.439945 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:53.440423 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:53.440643 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:53.440647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:53.441190 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:53.441369 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.441373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.442081 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.442082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.442870 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.442872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.443392 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.443393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.443532 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.444000 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:53.444008 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:53.445529 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:53.447062 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:53.447068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:53.447070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:53.447279 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:53.447475 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:53.447476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:53.447476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:53.447558 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:53.447831 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:53.447834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:53.447939 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 07:00:53.448041 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 07:00:53.448163 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260808 07:00:53.448983 17090 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45115
I20260808 07:00:53.449059 17090 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:53.449222 17090 heartbeater.cc:511] Master 127.21.28.126:45115 requested a full tablet report, sending...
I20260808 07:00:53.449683 16773 ts_manager.cc:194] Registered new tserver with Master: 5a5315c877de4e7e9b06ae955f0b481e (127.21.28.65:36157)
I20260808 07:00:53.450268 16773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:46719
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 07:00:53.466596 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45115
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:53.551998 17097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:53.552218 17097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:53.552242 17097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:53.553432 17097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:53.553458 17097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:53.553483 17097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 07:00:53.555068 17097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:53.555301 17097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:53.555516 17097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:53.560768 17110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:53.560889 17114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:53.560791 17106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:53.560921 17097 server_base.cc:1034] running on GCE node
I20260808 07:00:53.561105 17097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:53.561347 17097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:53.562506 17097 hybrid_clock.cc:648] HybridClock initialized: now 1786172453562490 us; error 33 us; skew 500 ppm
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:53.565384 17097 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 07:00:53.565871 17097 webserver.cc:460] Webserver started at http://127.21.28.66:35667/ using document root <none> and password file <none>
I20260808 07:00:53.566102 17097 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:53.566157 17097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:53.566294 17097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:53.567582 17097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "99e836d00e4144a687e57e1fd5f0924d"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "3c9ffe9b9971c860cb1e5c159b082844"
server_key_iv: "73e86f5f5b52fa623d5453eccad718a4"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.568001 17097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "99e836d00e4144a687e57e1fd5f0924d"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "3c9ffe9b9971c860cb1e5c159b082844"
server_key_iv: "73e86f5f5b52fa623d5453eccad718a4"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.569569 17097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:53.570384 17124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.570577 17097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:53.570660 17097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "99e836d00e4144a687e57e1fd5f0924d"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "3c9ffe9b9971c860cb1e5c159b082844"
server_key_iv: "73e86f5f5b52fa623d5453eccad718a4"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.570711 17097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:53.585634 17097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:53.588021 17097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:53.588163 17097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:53.588409 17097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:53.588733 17097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:53.588775 17097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.588809 17097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:53.588831 17097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.595574 17097 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:45065
I20260808 07:00:53.595613 17299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:45065 every 8 connection(s)
I20260808 07:00:53.596000 17097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:53.605378 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.596749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:57461 (local address 127.21.28.126:45115)
0808 07:00:53.596806 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:53.596808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:53.596999 (+ 191us) server_negotiation.cc:407] Connection header received
0808 07:00:53.597504 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:53.597505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:53.597517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:53.597576 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:53.598534 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.599310 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.600274 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.600573 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.600640 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.602849 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:53.602850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:53.602851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:53.602859 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:53.604250 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.604578 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.604579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.604580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.604638 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.604826 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.604827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.604827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.604956 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:53.605018 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:53.605206 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 07:00:53.605283 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 07:00:53.605451 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.596702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45115 (local address 127.21.28.66:57461)
0808 07:00:53.596889 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:53.596925 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:53.597414 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:53.597655 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:53.597659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:53.598222 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:53.598454 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.598459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.599385 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.599386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.600200 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.600202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.600639 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.600640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.600756 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.601195 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:53.601203 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:53.602765 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:53.604318 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:53.604323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:53.604325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:53.604518 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:53.604692 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:53.604692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:53.604693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:53.604781 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:53.605005 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:53.605008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:53.605106 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 07:00:53.605176 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 07:00:53.605290 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260808 07:00:53.605913 17300 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45115
I20260808 07:00:53.606005 17300 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:53.606081 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 17097
I20260808 07:00:53.606184 17300 heartbeater.cc:511] Master 127.21.28.126:45115 requested a full tablet report, sending...
I20260808 07:00:53.606194 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 07:00:53.606384 21617 external_mini_cluster.cc:1428] Setting key 16b5d4b1b35be24ae134763fb122026e
I20260808 07:00:53.606564 16773 ts_manager.cc:194] Registered new tserver with Master: 99e836d00e4144a687e57e1fd5f0924d (127.21.28.66:45065)
I20260808 07:00:53.607129 16773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:57461
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 07:00:53.630631 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45115
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:53.710616 17309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:53.710798 17309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:53.710827 17309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:53.711972 17309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:53.712011 17309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:53.712046 17309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 07:00:53.713486 17309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:45115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:53.713702 17309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:53.713896 17309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:53.718526 17320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:53.718581 17316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:53.718559 17317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:53.718642 17309 server_base.cc:1034] running on GCE node
I20260808 07:00:53.718855 17309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:53.719061 17309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:53.720191 17309 hybrid_clock.cc:648] HybridClock initialized: now 1786172453720175 us; error 26 us; skew 500 ppm
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:53.722621 17309 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 07:00:53.723019 17309 webserver.cc:460] Webserver started at http://127.21.28.67:36777/ using document root <none> and password file <none>
I20260808 07:00:53.723207 17309 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:53.723253 17309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:53.723371 17309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:53.724388 17309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "a14204b74a27499a894024a46093d28d"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "86488c178bd2d7c0a98acda1e881a8ca"
server_key_iv: "b05162e6a0c556612cabbe48db46bc3d"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.724709 17309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "a14204b74a27499a894024a46093d28d"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "86488c178bd2d7c0a98acda1e881a8ca"
server_key_iv: "b05162e6a0c556612cabbe48db46bc3d"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.725880 17309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:00:53.726495 17334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.726632 17309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:53.726714 17309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "a14204b74a27499a894024a46093d28d"
format_stamp: "Formatted at 2026-08-08 07:00:53 on dist-test-slave-nj21"
server_key: "86488c178bd2d7c0a98acda1e881a8ca"
server_key_iv: "b05162e6a0c556612cabbe48db46bc3d"
server_key_version: "encryptionkey@0"
I20260808 07:00:53.726768 17309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:53.752270 17309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:53.754089 17309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:53.754222 17309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:53.754462 17309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:53.754782 17309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:53.754815 17309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.754843 17309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:53.754868 17309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:53.760885 17309 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:36753
I20260808 07:00:53.760923 17519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:36753 every 8 connection(s)
I20260808 07:00:53.761216 17309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 07:00:53.764741 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 17309
I20260808 07:00:53.764853 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 07:00:53.765043 21617 external_mini_cluster.cc:1428] Setting key ac62a63da1f8fdea83a0e78bc2ab82e0
Aug 08 07:00:53 dist-test-slave-nj21 krb5kdc[16698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172453, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:53.769742 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.761815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:38467 (local address 127.21.28.126:45115)
0808 07:00:53.761857 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:53.761859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:53.762059 (+ 200us) server_negotiation.cc:407] Connection header received
0808 07:00:53.762501 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:53.762502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:53.762512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:53.762557 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:53.763439 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.763988 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.765103 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.765390 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.765452 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.767475 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:53.767476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:53.767476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:53.767482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:53.768732 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.769044 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.769044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.769045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.769097 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.769278 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.769279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.769279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.769370 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:53.769415 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:53.769625 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 07:00:53.769680 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:00:53.769887 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.761803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:45115 (local address 127.21.28.67:38467)
0808 07:00:53.761945 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:53.761977 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:53.762429 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:53.762614 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:53.762617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:53.763153 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:53.763343 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.763348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.764054 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.764054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.765020 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.765024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.765471 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.765472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.765620 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.766099 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:53.766107 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:53.767401 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:53.768793 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:53.768798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:53.768799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:53.768988 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:53.769143 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:53.769144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:53.769144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:53.769230 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:53.769412 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:53.769415 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:53.769530 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 07:00:53.769607 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 07:00:53.769715 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260808 07:00:53.770319 17521 heartbeater.cc:347] Connected to a master server at 127.21.28.126:45115
I20260808 07:00:53.770408 17521 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:53.770573 17521 heartbeater.cc:511] Master 127.21.28.126:45115 requested a full tablet report, sending...
I20260808 07:00:53.770910 16773 ts_manager.cc:194] Registered new tserver with Master: a14204b74a27499a894024a46093d28d (127.21.28.67:36753)
I20260808 07:00:53.771374 16773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:38467
I20260808 07:00:53.776513 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:00:53.783097 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.777938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52544 (local address 127.21.28.126:45115)
0808 07:00:53.777972 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:53.777973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:53.778085 (+ 112us) server_negotiation.cc:407] Connection header received
0808 07:00:53.778121 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:53.778122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:53.778129 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:53.778170 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:53.778648 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.779098 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.779796 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.779973 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.780020 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.781109 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:53.781109 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:53.781110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:53.781115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:53.782262 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.782560 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.782560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.782560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.782604 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:53.782729 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:53.782729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:53.782729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:53.782840 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:53.782911 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:53.782995 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 07:00:53.783040 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 07:00:53.784413 16773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:00:53.785871 16773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:00:53.791411 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.787952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:36753 (local address 127.0.0.1:42016)
0808 07:00:53.788033 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:53.788043 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:53.788104 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:53.788267 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:53.788269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:53.788271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:53.788335 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.788336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.789292 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.789293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.790287 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.790289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.791137 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.791137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.791213 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.791225 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 07:00:53.791289 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 07:00:53.791341 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":48,"threads_started":1}
I20260808 07:00:53.791421 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.787797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.21.28.67:36753)
0808 07:00:53.787958 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:53.787962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:53.788100 (+ 138us) server_negotiation.cc:407] Connection header received
0808 07:00:53.788167 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:53.788168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:53.788180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:53.788244 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:53.788403 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.789211 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.790372 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.791082 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.791137 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.791194 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:53.791312 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 07:00:53.791350 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260808 07:00:53.792141 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.787732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:36157 (local address 127.0.0.1:50424)
0808 07:00:53.787835 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:53.787877 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:53.787947 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:53.788625 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:53.788626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:53.788628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:53.788678 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.788679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.789956 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.789957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.790995 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.790997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.791898 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.791898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.791954 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.791957 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 07:00:53.792011 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 07:00:53.792051 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":38,"thread_start_us":52,"threads_started":1}
I20260808 07:00:53.792155 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.788092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.21.28.66:45065)
0808 07:00:53.788236 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:53.788239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:53.788393 (+ 154us) server_negotiation.cc:407] Connection header received
0808 07:00:53.788514 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:53.788516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:53.788531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:53.788611 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:53.788819 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.789883 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.790994 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.791724 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.791789 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.791843 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:53.792038 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 07:00:53.792082 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260808 07:00:53.792140 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.788250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:45065 (local address 127.0.0.1:60468)
0808 07:00:53.788331 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:53.788340 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:53.788420 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:53.788689 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:53.788690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:53.788692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:53.788731 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.788732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.789968 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.789969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.790921 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:53.790923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.791879 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.791880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:53.791954 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.791959 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 07:00:53.792011 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 07:00:53.792051 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":25,"thread_start_us":54,"threads_started":1}
I20260808 07:00:53.792291 17416 tablet_service.cc:1467] Processing CreateTablet for tablet e534acabb1c14353aa8ad646823dcc8f (DEFAULT_TABLE table=test-table [id=e3fdd81b7693449c8a7cab346ad5ca27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:53.792557 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:53.787892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.21.28.65:36157)
0808 07:00:53.788396 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:53.788400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:53.788409 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:53.788446 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:53.788448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:53.788462 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:53.788549 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:53.788746 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.789844 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.791066 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:53.791773 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:53.791825 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:53.791871 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:53.792417 (+ 546us) server_negotiation.cc:299] Negotiation successful
0808 07:00:53.792483 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":81,"threads_started":1}
I20260808 07:00:53.792683 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e534acabb1c14353aa8ad646823dcc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:53.792909 17198 tablet_service.cc:1467] Processing CreateTablet for tablet e534acabb1c14353aa8ad646823dcc8f (DEFAULT_TABLE table=test-table [id=e3fdd81b7693449c8a7cab346ad5ca27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:53.793241 17198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e534acabb1c14353aa8ad646823dcc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:53.795142 17542 tablet_bootstrap.cc:492] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Bootstrap starting.
I20260808 07:00:53.795426 16976 tablet_service.cc:1467] Processing CreateTablet for tablet e534acabb1c14353aa8ad646823dcc8f (DEFAULT_TABLE table=test-table [id=e3fdd81b7693449c8a7cab346ad5ca27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:53.795778 16976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e534acabb1c14353aa8ad646823dcc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:53.795917 17547 tablet_bootstrap.cc:492] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Bootstrap starting.
I20260808 07:00:53.795964 17542 tablet_bootstrap.cc:654] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:53.796530 17542 log.cc:824] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:53.796581 17547 tablet_bootstrap.cc:654] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:53.797065 17547 log.cc:824] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:53.797353 17542 tablet_bootstrap.cc:492] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: No bootstrap required, opened a new log
I20260808 07:00:53.797415 17542 ts_tablet_manager.cc:1397] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:53.797778 17547 tablet_bootstrap.cc:492] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: No bootstrap required, opened a new log
I20260808 07:00:53.797844 17547 ts_tablet_manager.cc:1397] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:53.798437 17550 tablet_bootstrap.cc:492] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e: Bootstrap starting.
I20260808 07:00:53.799182 17550 tablet_bootstrap.cc:654] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:53.799315 17542 raft_consensus.cc:348] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:53.799489 17542 raft_consensus.cc:374] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:53.799526 17542 raft_consensus.cc:729] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14204b74a27499a894024a46093d28d, State: Initialized, Role: FOLLOWER
I20260808 07:00:53.799660 17542 consensus_queue.cc:260] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:53.799747 17550 log.cc:824] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:53.799834 17547 raft_consensus.cc:348] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:53.799970 17542 ts_tablet_manager.cc:1428] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260808 07:00:53.800005 17547 raft_consensus.cc:374] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:53.800058 17547 raft_consensus.cc:729] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e836d00e4144a687e57e1fd5f0924d, State: Initialized, Role: FOLLOWER
I20260808 07:00:53.800076 17521 heartbeater.cc:503] Master 127.21.28.126:45115 was elected leader, sending a full tablet report...
I20260808 07:00:53.800175 17547 consensus_queue.cc:260] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:53.800442 17300 heartbeater.cc:503] Master 127.21.28.126:45115 was elected leader, sending a full tablet report...
I20260808 07:00:53.800457 17547 ts_tablet_manager.cc:1428] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:53.800585 17550 tablet_bootstrap.cc:492] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e: No bootstrap required, opened a new log
I20260808 07:00:53.800658 17550 ts_tablet_manager.cc:1397] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:53.802212 17550 raft_consensus.cc:348] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:53.802337 17550 raft_consensus.cc:374] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:53.802358 17550 raft_consensus.cc:729] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5315c877de4e7e9b06ae955f0b481e, State: Initialized, Role: FOLLOWER
I20260808 07:00:53.802459 17550 consensus_queue.cc:260] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:53.802675 17550 ts_tablet_manager.cc:1428] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:53.802708 17090 heartbeater.cc:503] Master 127.21.28.126:45115 was elected leader, sending a full tablet report...
W20260808 07:00:53.846596 17301 tablet.cc:2369] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:00:53.939738 17091 tablet.cc:2369] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:00:54.011826 17522 tablet.cc:2369] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:54.149501 17554 raft_consensus.cc:482] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:54.149629 17554 raft_consensus.cc:504] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:54.149927 17554 leader_election.cc:288] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a14204b74a27499a894024a46093d28d (127.21.28.67:36753), 99e836d00e4144a687e57e1fd5f0924d (127.21.28.66:45065)
I20260808 07:00:54.153676 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.150174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:42771 (local address 127.21.28.66:45065)
0808 07:00:54.150247 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.150248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.150383 (+ 135us) server_negotiation.cc:407] Connection header received
0808 07:00:54.150427 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.150428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.150438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.150486 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:54.150627 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.151338 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.152782 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.153327 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.153399 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.153415 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.153569 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.153604 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 07:00:54.153676 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.150197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:45411 (local address 127.21.28.67:36753)
0808 07:00:54.150240 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.150242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.150286 (+ 44us) server_negotiation.cc:407] Connection header received
0808 07:00:54.150350 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.150351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.150359 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.150422 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:54.150593 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.151343 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.152782 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.153345 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.153405 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.153414 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.153569 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.153604 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 07:00:54.153702 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.150085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:36753 (local address 127.21.28.65:45411)
0808 07:00:54.150224 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.150236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.150298 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.150436 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.150437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.150439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:54.150519 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.150521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.151414 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.151415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.152695 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.152699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.153410 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.153411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.153498 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.153504 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.153557 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.153600 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":53,"thread_start_us":66,"threads_started":1}
I20260808 07:00:54.153707 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.150085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:45065 (local address 127.21.28.65:42771)
0808 07:00:54.150317 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.150326 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.150375 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.150497 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.150498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.150500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:54.150568 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.150570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.151416 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.151416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.152695 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.152699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.153412 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.153412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.153498 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.153505 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.153557 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.153600 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":40,"thread_start_us":55,"threads_started":1}
I20260808 07:00:54.154034 17444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e534acabb1c14353aa8ad646823dcc8f" candidate_uuid: "5a5315c877de4e7e9b06ae955f0b481e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14204b74a27499a894024a46093d28d" is_pre_election: true
I20260808 07:00:54.154186 17444 raft_consensus.cc:2454] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a5315c877de4e7e9b06ae955f0b481e in term 0.
I20260808 07:00:54.154201 17226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e534acabb1c14353aa8ad646823dcc8f" candidate_uuid: "5a5315c877de4e7e9b06ae955f0b481e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e836d00e4144a687e57e1fd5f0924d" is_pre_election: true
I20260808 07:00:54.154346 17226 raft_consensus.cc:2454] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a5315c877de4e7e9b06ae955f0b481e in term 0.
I20260808 07:00:54.154419 16902 leader_election.cc:302] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5315c877de4e7e9b06ae955f0b481e, a14204b74a27499a894024a46093d28d; no voters:
I20260808 07:00:54.154593 17554 raft_consensus.cc:2781] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:00:54.154644 17554 raft_consensus.cc:482] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:00:54.154659 17554 raft_consensus.cc:3037] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:54.155377 17554 raft_consensus.cc:504] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:54.155504 17554 leader_election.cc:288] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [CANDIDATE]: Term 1 election: Requested vote from peers a14204b74a27499a894024a46093d28d (127.21.28.67:36753), 99e836d00e4144a687e57e1fd5f0924d (127.21.28.66:45065)
I20260808 07:00:54.155642 17444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e534acabb1c14353aa8ad646823dcc8f" candidate_uuid: "5a5315c877de4e7e9b06ae955f0b481e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14204b74a27499a894024a46093d28d"
I20260808 07:00:54.155649 17226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e534acabb1c14353aa8ad646823dcc8f" candidate_uuid: "5a5315c877de4e7e9b06ae955f0b481e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e836d00e4144a687e57e1fd5f0924d"
I20260808 07:00:54.155741 17226 raft_consensus.cc:3037] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:54.155740 17444 raft_consensus.cc:3037] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:54.156441 17444 raft_consensus.cc:2454] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a5315c877de4e7e9b06ae955f0b481e in term 1.
I20260808 07:00:54.156482 17226 raft_consensus.cc:2454] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a5315c877de4e7e9b06ae955f0b481e in term 1.
I20260808 07:00:54.156591 16902 leader_election.cc:302] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5315c877de4e7e9b06ae955f0b481e, a14204b74a27499a894024a46093d28d; no voters:
I20260808 07:00:54.156692 17554 raft_consensus.cc:2781] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:54.156831 17554 raft_consensus.cc:686] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 1 LEADER]: Becoming Leader. State: Replica: 5a5315c877de4e7e9b06ae955f0b481e, State: Running, Role: LEADER
I20260808 07:00:54.156922 17554 consensus_queue.cc:237] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } }
I20260808 07:00:54.157666 16773 catalog_manager.cc:5495] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a5315c877de4e7e9b06ae955f0b481e (127.21.28.65). New cstate: current_term: 1 leader_uuid: "5a5315c877de4e7e9b06ae955f0b481e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a5315c877de4e7e9b06ae955f0b481e" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 36157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 } health_report { overall_health: UNKNOWN } } }
I20260808 07:00:54.229333 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.227063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.21.28.65:36157)
0808 07:00:54.227107 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.227109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.227118 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:54.227176 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.227178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.227198 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.227257 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:00:54.227391 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.227885 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.228595 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.228807 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.228865 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.229029 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:00:54.229112 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:00:54.229195 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.229253 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.229264 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:00:54.231055 17444 raft_consensus.cc:1261] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 1 FOLLOWER]: Refusing update from remote peer 5a5315c877de4e7e9b06ae955f0b481e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:54.231396 17226 raft_consensus.cc:1261] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 1 FOLLOWER]: Refusing update from remote peer 5a5315c877de4e7e9b06ae955f0b481e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:54.231377 17560 consensus_queue.cc:1035] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14204b74a27499a894024a46093d28d" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:54.231655 17560 consensus_queue.cc:1035] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e836d00e4144a687e57e1fd5f0924d" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:54.233783 17568 mvcc.cc:205] Tried to move back new op lower bound from 7316162372527730688 to 7316162372227231744. Current Snapshot: MvccSnapshot[applied={T|T < 7316162372527730688}]
I20260808 07:00:54.235968 16768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:54.236066 16768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:54.236785 16768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0a291e314e6474b9e702f6412b98e96: Sending DeleteTablet for 3 replicas of tablet e534acabb1c14353aa8ad646823dcc8f
I20260808 07:00:54.237109 17416 tablet_service.cc:1514] Processing DeleteTablet for tablet e534acabb1c14353aa8ad646823dcc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:54 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:42016
I20260808 07:00:54.237129 16976 tablet_service.cc:1514] Processing DeleteTablet for tablet e534acabb1c14353aa8ad646823dcc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:54 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:50424
I20260808 07:00:54.237432 17198 tablet_service.cc:1514] Processing DeleteTablet for tablet e534acabb1c14353aa8ad646823dcc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:54 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:60468
I20260808 07:00:54.237555 17584 tablet_replica.cc:330] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e: stopping tablet replica
I20260808 07:00:54.237656 17584 raft_consensus.cc:2229] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:00:54.237661 17585 tablet_replica.cc:330] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: stopping tablet replica
I20260808 07:00:54.237747 17584 raft_consensus.cc:2258] T e534acabb1c14353aa8ad646823dcc8f P 5a5315c877de4e7e9b06ae955f0b481e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:54.237805 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 16877
I20260808 07:00:54.240535 17585 raft_consensus.cc:2229] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:54.240731 17585 raft_consensus.cc:2258] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:54.244014 17585 ts_tablet_manager.cc:1905] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:54.245779 17585 ts_tablet_manager.cc:1918] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:54.245843 17585 log.cc:1196] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/e534acabb1c14353aa8ad646823dcc8f
I20260808 07:00:54.246075 17585 ts_tablet_manager.cc:1939] T e534acabb1c14353aa8ad646823dcc8f P 99e836d00e4144a687e57e1fd5f0924d: Deleting consensus metadata
I20260808 07:00:54.246439 16740 catalog_manager.cc:4841] TS 99e836d00e4144a687e57e1fd5f0924d (127.21.28.66:45065): tablet e534acabb1c14353aa8ad646823dcc8f (table test-table [id=e3fdd81b7693449c8a7cab346ad5ca27]) successfully deleted
I20260808 07:00:54.247545 17582 tablet_replica.cc:330] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: stopping tablet replica
I20260808 07:00:54.247658 17582 raft_consensus.cc:2229] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:54.247774 17582 raft_consensus.cc:2258] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 07:00:54.251418 16738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:36157 (error 108)
I20260808 07:00:54.251725 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 17097
I20260808 07:00:54.251892 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.251778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:36157 (local address 127.0.0.1:50454)
0808 07:00:54.251827 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.251861 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 07:00:54.251775 17582 ts_tablet_manager.cc:1905] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:54.254755 17582 ts_tablet_manager.cc:1918] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:54.254827 17582 log.cc:1196] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/e534acabb1c14353aa8ad646823dcc8f
I20260808 07:00:54.255082 17582 ts_tablet_manager.cc:1939] T e534acabb1c14353aa8ad646823dcc8f P a14204b74a27499a894024a46093d28d: Deleting consensus metadata
I20260808 07:00:54.255478 16742 catalog_manager.cc:4841] TS a14204b74a27499a894024a46093d28d (127.21.28.67:36753): tablet e534acabb1c14353aa8ad646823dcc8f (table test-table [id=e3fdd81b7693449c8a7cab346ad5ca27]) successfully deleted
W20260808 07:00:54.255559 16738 catalog_manager.cc:4584] TS 5a5315c877de4e7e9b06ae955f0b481e (127.21.28.65:36157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e534acabb1c14353aa8ad646823dcc8f: Network error: Client connection negotiation failed: client connection to 127.21.28.65:36157: connect: Connection refused (error 111)
I20260808 07:00:54.262719 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 17309
I20260808 07:00:54.271386 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 16717
2026-08-08T07:00:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2350 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T07:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:00:54Z Disabled control of system clock
I20260808 07:00:54.304382 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:34345
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:34345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:54.382265 17592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:54.382464 17592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:54.382483 17592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:54.383584 17592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:54.383625 17592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:54.383636 17592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:54.383659 17592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:54.383693 17592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:54.385003 17592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:34345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:54.385215 17592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:54.385468 17592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:54.389806 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:54.389869 17603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:54.389838 17600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:54.389962 17592 server_base.cc:1034] running on GCE node
I20260808 07:00:54.390249 17592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:54.390504 17592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:54.391638 17592 hybrid_clock.cc:648] HybridClock initialized: now 1786172454391628 us; error 30 us; skew 500 ppm
I20260808 07:00:54.395371 17592 webserver.cc:460] Webserver started at http://127.21.28.126:33747/ using document root <none> and password file <none>
I20260808 07:00:54.395576 17592 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:54.395622 17592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:54.395764 17592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:54.396746 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "0a4ef61061a34dd6b9f80d2d72599372"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "b3e918ad2c7346afdda09dff4dbec29c"
server_key_iv: "786fd726c9a6483bf4ddb79846e4cfd9"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.397063 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "0a4ef61061a34dd6b9f80d2d72599372"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "b3e918ad2c7346afdda09dff4dbec29c"
server_key_iv: "786fd726c9a6483bf4ddb79846e4cfd9"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.398191 17592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 07:00:54.398795 17616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.398948 17592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.399027 17592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "0a4ef61061a34dd6b9f80d2d72599372"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "b3e918ad2c7346afdda09dff4dbec29c"
server_key_iv: "786fd726c9a6483bf4ddb79846e4cfd9"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.399084 17592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:54.413468 17592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:54.413702 17592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:54.413821 17592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:54.417372 17592 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:34345
I20260808 07:00:54.417412 17712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:34345 every 8 connection(s)
I20260808 07:00:54.417670 17592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:54.418071 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 17592
I20260808 07:00:54.418186 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:00:54.418380 21617 external_mini_cluster.cc:1428] Setting key 99c3328706596c85f78ab7d56794e8b6
I20260808 07:00:54.418737 17713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:54.421300 17713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372: Bootstrap starting.
I20260808 07:00:54.421841 17713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:54.422092 17713 log.cc:824] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:54.423003 17713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372: No bootstrap required, opened a new log
I20260808 07:00:54.423502 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.418864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.21.28.126:34345)
0808 07:00:54.419046 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.419055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.419106 (+ 51us) server_negotiation.cc:407] Connection header received
0808 07:00:54.419624 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.419630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.419711 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.419837 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:54.420049 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.420908 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.421729 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.422158 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.422260 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.422843 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:54.422847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:54.422848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:54.423031 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:54.423071 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:54.423085 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:54.423153 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.423210 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.423287 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260808 07:00:54.424849 17713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4ef61061a34dd6b9f80d2d72599372" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34345 } }
I20260808 07:00:54.424980 17713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:54.425002 17713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4ef61061a34dd6b9f80d2d72599372, State: Initialized, Role: FOLLOWER
I20260808 07:00:54.425109 17713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4ef61061a34dd6b9f80d2d72599372" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34345 } }
I20260808 07:00:54.425170 17713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:54.425201 17713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:54.425246 17713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:54.425735 17713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4ef61061a34dd6b9f80d2d72599372" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34345 } }
I20260808 07:00:54.425822 17713 leader_election.cc:302] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a4ef61061a34dd6b9f80d2d72599372; no voters:
I20260808 07:00:54.425990 17713 leader_election.cc:288] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:00:54.426038 17720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:54.426186 17720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [term 1 LEADER]: Becoming Leader. State: Replica: 0a4ef61061a34dd6b9f80d2d72599372, State: Running, Role: LEADER
I20260808 07:00:54.426226 17713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:54.426271 17720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4ef61061a34dd6b9f80d2d72599372" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34345 } }
I20260808 07:00:54.426753 17721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a4ef61061a34dd6b9f80d2d72599372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4ef61061a34dd6b9f80d2d72599372" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34345 } } }
I20260808 07:00:54.426868 17721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:54.426756 17722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a4ef61061a34dd6b9f80d2d72599372. Latest consensus state: current_term: 1 leader_uuid: "0a4ef61061a34dd6b9f80d2d72599372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4ef61061a34dd6b9f80d2d72599372" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 34345 } } }
I20260808 07:00:54.427031 17722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:54.427289 17735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:54.427773 17735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:54.429013 17735 catalog_manager.cc:1347] Generated new cluster ID: 27e0d9180ac145f197644e632b7e5dfc
I20260808 07:00:54.429064 17735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:00:54.460688 17735 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:00:54.461182 17735 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:54.468712 17735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372: Generated new TSK 0
I20260808 07:00:54.468863 17735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:00:54.478789 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:34345
--builtin_ntp_servers=127.21.28.84:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:54.557771 17752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:54.557951 17752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:54.557971 17752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:54.559062 17752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:54.559099 17752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:54.559124 17752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 07:00:54.560444 17752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:54.560654 17752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:54.560910 17752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:54.565392 17763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:54.565369 17760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:54.565541 17752 server_base.cc:1034] running on GCE node
W20260808 07:00:54.565439 17767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:54.565829 17752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:54.566049 17752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:54.567185 17752 hybrid_clock.cc:648] HybridClock initialized: now 1786172454567176 us; error 30 us; skew 500 ppm
I20260808 07:00:54.568248 17752 webserver.cc:460] Webserver started at http://127.21.28.65:32877/ using document root <none> and password file <none>
I20260808 07:00:54.568461 17752 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:54.568511 17752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:54.568634 17752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:54.569586 17752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "6e99a915b92c45dcb0b957ea0ff22421"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "5d706e7c8a6b0b6b655869e4f7992320"
server_key_iv: "bb7ed80b8f1e19e92bea94f427ad7a24"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.569890 17752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "6e99a915b92c45dcb0b957ea0ff22421"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "5d706e7c8a6b0b6b655869e4f7992320"
server_key_iv: "bb7ed80b8f1e19e92bea94f427ad7a24"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.571010 17752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:00:54.571588 17779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.571744 17752 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:00:54.571825 17752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "6e99a915b92c45dcb0b957ea0ff22421"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "5d706e7c8a6b0b6b655869e4f7992320"
server_key_iv: "bb7ed80b8f1e19e92bea94f427ad7a24"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.571882 17752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:54.580279 17752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:54.580500 17752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:54.580612 17752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:54.580788 17752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:54.581058 17752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:54.581089 17752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.581113 17752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:54.581138 17752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.586362 17752 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:43583
I20260808 07:00:54.586400 17967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:43583 every 8 connection(s)
I20260808 07:00:54.586652 17752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 07:00:54.591235 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.587250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:50005 (local address 127.21.28.126:34345)
0808 07:00:54.587296 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.587298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.587516 (+ 218us) server_negotiation.cc:407] Connection header received
0808 07:00:54.587998 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.587999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.588015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.588095 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:54.588693 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.589191 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.590102 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.590289 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.590342 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.590982 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:54.590983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:54.590984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:54.590988 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:54.591004 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:54.591008 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:54.591049 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.591122 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.591151 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 07:00:54.591392 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.587228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:34345 (local address 127.21.28.65:50005)
0808 07:00:54.587414 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.587443 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.587910 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.588141 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.588144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.588154 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:54.588612 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.588616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.589271 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.589272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.590024 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.590026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.590359 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.590360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.590484 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.590866 (+ 382us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:54.590867 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:54.590890 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:54.590910 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:54.591055 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:54.591061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:54.591064 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.591120 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.591210 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260808 07:00:54.591915 17968 heartbeater.cc:347] Connected to a master server at 127.21.28.126:34345
I20260808 07:00:54.591996 17968 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:54.592127 17968 heartbeater.cc:511] Master 127.21.28.126:34345 requested a full tablet report, sending...
I20260808 07:00:54.592546 17647 ts_manager.cc:194] Registered new tserver with Master: 6e99a915b92c45dcb0b957ea0ff22421 (127.21.28.65:43583)
I20260808 07:00:54.593135 17647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.65:50005
I20260808 07:00:54.593564 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 17752
I20260808 07:00:54.593658 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 07:00:54.593837 21617 external_mini_cluster.cc:1428] Setting key 775a4456a04121414f7243ceddb3090a
I20260808 07:00:54.594591 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:34345
--builtin_ntp_servers=127.21.28.84:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:54.672231 17973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:54.672410 17973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:54.672437 17973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:54.673512 17973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:54.673549 17973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:54.673581 17973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 07:00:54.675007 17973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:54.675241 17973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:54.675506 17973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:54.680136 17986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:54.680194 17988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:54.680147 17985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:54.680472 17973 server_base.cc:1034] running on GCE node
I20260808 07:00:54.680675 17973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:54.680857 17973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:54.681986 17973 hybrid_clock.cc:648] HybridClock initialized: now 1786172454681974 us; error 27 us; skew 500 ppm
I20260808 07:00:54.683140 17973 webserver.cc:460] Webserver started at http://127.21.28.66:38257/ using document root <none> and password file <none>
I20260808 07:00:54.683363 17973 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:54.683411 17973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:54.683534 17973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:54.684576 17973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "ff2fcdb884a24d0db994610a7a01d8df"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "a52496a6656f321d40a3f3598fe4eafe"
server_key_iv: "2a393dd9bb4cc30aa0982b3cf29746c4"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.684886 17973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "ff2fcdb884a24d0db994610a7a01d8df"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "a52496a6656f321d40a3f3598fe4eafe"
server_key_iv: "2a393dd9bb4cc30aa0982b3cf29746c4"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.686017 17973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:54.686640 17997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.686770 17973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:54.686851 17973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "ff2fcdb884a24d0db994610a7a01d8df"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "a52496a6656f321d40a3f3598fe4eafe"
server_key_iv: "2a393dd9bb4cc30aa0982b3cf29746c4"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.686919 17973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:54.694530 17973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:54.694756 17973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:54.694870 17973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:54.695060 17973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:54.695336 17973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:54.695369 17973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.695394 17973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:54.695411 17973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.701001 17973 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:45529
I20260808 07:00:54.701051 18171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:45529 every 8 connection(s)
I20260808 07:00:54.701325 17973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 07:00:54.705713 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.701956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:44125 (local address 127.21.28.126:34345)
0808 07:00:54.702002 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.702004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.702206 (+ 202us) server_negotiation.cc:407] Connection header received
0808 07:00:54.702633 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.702634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.702643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.702706 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:54.703313 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.703834 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.704710 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.704882 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.704928 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.705527 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:54.705528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:54.705528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:54.705532 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:54.705546 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:54.705550 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:54.705585 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.705631 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.705652 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 07:00:54.705930 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.701947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:34345 (local address 127.21.28.66:44125)
0808 07:00:54.702105 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.702136 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.702566 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.702784 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.702787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.702795 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:54.703248 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.703252 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.703893 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.703894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.704655 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.704657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.704932 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.704933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.705027 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.705420 (+ 393us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:54.705420 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:54.705444 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:54.705463 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:54.705587 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:54.705589 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:54.705591 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.705631 (+ 40us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.705708 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260808 07:00:54.706333 18172 heartbeater.cc:347] Connected to a master server at 127.21.28.126:34345
I20260808 07:00:54.706413 18172 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:54.706568 18172 heartbeater.cc:511] Master 127.21.28.126:34345 requested a full tablet report, sending...
I20260808 07:00:54.706874 17647 ts_manager.cc:194] Registered new tserver with Master: ff2fcdb884a24d0db994610a7a01d8df (127.21.28.66:45529)
I20260808 07:00:54.707218 17647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.66:44125
I20260808 07:00:54.709041 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 17973
I20260808 07:00:54.709131 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 07:00:54.709295 21617 external_mini_cluster.cc:1428] Setting key 8f0ebc8c4f4518376a89d973a5cec0d4
I20260808 07:00:54.710080 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:34345
--builtin_ntp_servers=127.21.28.84:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:54.788513 18179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:54.788699 18179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:54.788718 18179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:54.789826 18179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:54.789862 18179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:54.789886 18179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 07:00:54.791208 18179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:54.791388 18179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:54.791621 18179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 07:00:54.796286 18179 server_base.cc:1034] running on GCE node
W20260808 07:00:54.796339 18187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:54.796270 18184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:54.796262 18185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:54.796686 18179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:54.796898 18179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:54.798036 18179 hybrid_clock.cc:648] HybridClock initialized: now 1786172454798022 us; error 28 us; skew 500 ppm
I20260808 07:00:54.799163 18179 webserver.cc:460] Webserver started at http://127.21.28.67:34259/ using document root <none> and password file <none>
I20260808 07:00:54.799376 18179 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:54.799430 18179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:54.799554 18179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:54.800657 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "8875135098ad4651aea91b765ebf2aa6"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "05248c10cf22f1a14610762819f3b195"
server_key_iv: "15e28019f33d2ce47c5f2677aab1d319"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.800987 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "8875135098ad4651aea91b765ebf2aa6"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "05248c10cf22f1a14610762819f3b195"
server_key_iv: "15e28019f33d2ce47c5f2677aab1d319"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.802260 18179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:54.802899 18196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.803033 18179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:54.803114 18179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "8875135098ad4651aea91b765ebf2aa6"
format_stamp: "Formatted at 2026-08-08 07:00:54 on dist-test-slave-nj21"
server_key: "05248c10cf22f1a14610762819f3b195"
server_key_iv: "15e28019f33d2ce47c5f2677aab1d319"
server_key_version: "encryptionkey@0"
I20260808 07:00:54.803172 18179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:54.813433 18179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:54.813704 18179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:54.813836 18179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:54.814029 18179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:54.814325 18179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:54.814364 18179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.814391 18179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:54.814425 18179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:54.820233 18179 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:39787
I20260808 07:00:54.820290 18387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:39787 every 8 connection(s)
I20260808 07:00:54.820565 18179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 07:00:54.824553 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 18179
I20260808 07:00:54.824646 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 07:00:54.824818 21617 external_mini_cluster.cc:1428] Setting key 2f0ea63ae508db8b6c3a5c0233d99bbf
I20260808 07:00:54.825273 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.821227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:58467 (local address 127.21.28.126:34345)
0808 07:00:54.821280 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.821282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.821501 (+ 219us) server_negotiation.cc:407] Connection header received
0808 07:00:54.821952 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.821953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.821963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.822011 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:54.822609 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.823092 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.824031 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.824204 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.824249 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.824997 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:54.824999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:54.824999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:54.825004 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:54.825022 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:54.825026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:54.825071 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.825165 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.825192 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:00:54.825457 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.821209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:34345 (local address 127.21.28.67:58467)
0808 07:00:54.821390 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.821425 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.821874 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.822061 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.822063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.822071 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:54.822547 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.822551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.823149 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.823149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.823975 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.823978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.824255 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.824256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.824350 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.824856 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:54.824857 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:54.824885 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:54.824922 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:54.825085 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:54.825090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:54.825092 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.825152 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.825251 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260808 07:00:54.825896 18389 heartbeater.cc:347] Connected to a master server at 127.21.28.126:34345
I20260808 07:00:54.825980 18389 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:54.826148 18389 heartbeater.cc:511] Master 127.21.28.126:34345 requested a full tablet report, sending...
I20260808 07:00:54.826506 17647 ts_manager.cc:194] Registered new tserver with Master: 8875135098ad4651aea91b765ebf2aa6 (127.21.28.67:39787)
I20260808 07:00:54.826910 17647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.67:58467
I20260808 07:00:54.836469 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:00:54.840878 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.837832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.21.28.126:34345)
0808 07:00:54.837892 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.837894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.837973 (+ 79us) server_negotiation.cc:407] Connection header received
0808 07:00:54.838032 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.838033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.838042 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.838094 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:54.838263 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.838733 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.839658 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.839959 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.840016 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.840615 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:54.840616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:54.840616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:54.840621 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:54.840634 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:54.840638 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:54.840717 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.840788 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.840798 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 07:00:54.842198 17647 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:00:54.843724 17647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:00:54.849395 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.845555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:45529 (local address 127.0.0.1:35026)
0808 07:00:54.845718 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.845746 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.845809 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.846205 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.846208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.846210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:54.846269 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.846271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.847217 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.847218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.848328 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.848330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.849124 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.849125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.849199 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.849211 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.849275 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.849323 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260808 07:00:54.849416 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.845852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.21.28.66:45529)
0808 07:00:54.846044 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.846048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.846059 (+ 11us) server_negotiation.cc:407] Connection header received
0808 07:00:54.846101 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.846104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.846120 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.846204 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:54.846341 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.847139 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.848430 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.849054 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.849114 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.849202 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.849288 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.849328 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260808 07:00:54.849726 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.845760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:43583 (local address 127.0.0.1:55994)
0808 07:00:54.846039 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.846048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.846086 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.846277 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.846278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.846280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:54.846324 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.846325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.847594 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.847595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.848571 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.848573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.849493 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.849494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.849561 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.849565 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.849619 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.849656 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":32,"thread_start_us":66,"threads_started":1}
I20260808 07:00:54.849778 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.845884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.21.28.65:43583)
0808 07:00:54.846082 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.846085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.846095 (+ 10us) server_negotiation.cc:407] Connection header received
0808 07:00:54.846135 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.846136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.846151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.846215 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:54.846389 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.847496 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.848630 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.849284 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.849350 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.849453 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.849651 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.849699 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260808 07:00:54.850059 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.845560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:39787 (local address 127.0.0.1:57494)
0808 07:00:54.845960 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.845970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.846022 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.846256 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.846257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.846259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:54.846288 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.846289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.847601 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.847602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.848736 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.848738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.849893 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.849904 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.849959 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.849961 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.849989 (+ 28us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.850016 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":37,"thread_start_us":53,"threads_started":1}
I20260808 07:00:54.850271 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.845677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.21.28.67:39787)
0808 07:00:54.845906 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.845911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.846075 (+ 164us) server_negotiation.cc:407] Connection header received
0808 07:00:54.846115 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.846117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.846131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.846200 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:54.846354 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.847527 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.848812 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.849635 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.849689 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.850133 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.850166 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.850207 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260808 07:00:54.850158 18065 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb80467a06543ce84cbc734acad822c (DEFAULT_TABLE table=test-table [id=0f41706b5bb540f0b201470ad7734be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:54.850481 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb80467a06543ce84cbc734acad822c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:54.850857 18282 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb80467a06543ce84cbc734acad822c (DEFAULT_TABLE table=test-table [id=0f41706b5bb540f0b201470ad7734be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:54.851207 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb80467a06543ce84cbc734acad822c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:54.852243 17858 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb80467a06543ce84cbc734acad822c (DEFAULT_TABLE table=test-table [id=0f41706b5bb540f0b201470ad7734be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:54.852542 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb80467a06543ce84cbc734acad822c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:54.853809 18410 tablet_bootstrap.cc:492] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: Bootstrap starting.
I20260808 07:00:54.854281 18412 tablet_bootstrap.cc:492] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: Bootstrap starting.
I20260808 07:00:54.854573 18410 tablet_bootstrap.cc:654] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:54.854794 18412 tablet_bootstrap.cc:654] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:54.855011 18412 log.cc:824] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:54.855018 18410 log.cc:824] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:54.855468 18416 tablet_bootstrap.cc:492] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421: Bootstrap starting.
I20260808 07:00:54.855643 18412 tablet_bootstrap.cc:492] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: No bootstrap required, opened a new log
I20260808 07:00:54.855731 18412 ts_tablet_manager.cc:1397] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:54.856037 18416 tablet_bootstrap.cc:654] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:54.856277 18416 log.cc:824] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:54.856902 18416 tablet_bootstrap.cc:492] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421: No bootstrap required, opened a new log
I20260808 07:00:54.856957 18416 ts_tablet_manager.cc:1397] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 07:00:54.857661 18412 raft_consensus.cc:348] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.857784 18412 raft_consensus.cc:374] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:54.857803 18412 raft_consensus.cc:729] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8875135098ad4651aea91b765ebf2aa6, State: Initialized, Role: FOLLOWER
I20260808 07:00:54.857898 18412 consensus_queue.cc:260] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.858134 18412 ts_tablet_manager.cc:1428] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:54.858180 18389 heartbeater.cc:503] Master 127.21.28.126:34345 was elected leader, sending a full tablet report...
I20260808 07:00:54.858671 18410 tablet_bootstrap.cc:492] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: No bootstrap required, opened a new log
I20260808 07:00:54.858718 18410 ts_tablet_manager.cc:1397] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 07:00:54.859131 18416 raft_consensus.cc:348] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.859274 18416 raft_consensus.cc:374] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:54.859313 18416 raft_consensus.cc:729] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e99a915b92c45dcb0b957ea0ff22421, State: Initialized, Role: FOLLOWER
I20260808 07:00:54.859419 18416 consensus_queue.cc:260] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.859652 18416 ts_tablet_manager.cc:1428] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 07:00:54.859709 17968 heartbeater.cc:503] Master 127.21.28.126:34345 was elected leader, sending a full tablet report...
I20260808 07:00:54.860769 18410 raft_consensus.cc:348] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.860884 18410 raft_consensus.cc:374] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:54.860911 18410 raft_consensus.cc:729] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2fcdb884a24d0db994610a7a01d8df, State: Initialized, Role: FOLLOWER
I20260808 07:00:54.860991 18410 consensus_queue.cc:260] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.861243 18410 ts_tablet_manager.cc:1428] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:54.861276 18172 heartbeater.cc:503] Master 127.21.28.126:34345 was elected leader, sending a full tablet report...
I20260808 07:00:54.880831 18421 raft_consensus.cc:482] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:54.880918 18421 raft_consensus.cc:504] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.881171 18421 leader_election.cc:288] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff2fcdb884a24d0db994610a7a01d8df (127.21.28.66:45529), 8875135098ad4651aea91b765ebf2aa6 (127.21.28.67:39787)
I20260808 07:00:54.883780 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.881389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:48147 (local address 127.21.28.67:39787)
0808 07:00:54.881439 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.881441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.881556 (+ 115us) server_negotiation.cc:407] Connection header received
0808 07:00:54.881592 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.881593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.881602 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.881654 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:54.881776 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.882210 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.883100 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.883451 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.883503 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.883511 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.883652 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.883712 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 07:00:54.883798 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.881333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:52249 (local address 127.21.28.66:45529)
0808 07:00:54.881382 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.881384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.881392 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:00:54.881450 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.881451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.881460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.881511 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:54.881661 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.882152 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.883091 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.883475 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.883539 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.883546 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.883705 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.883743 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 07:00:54.883786 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.881328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:39787 (local address 127.21.28.65:48147)
0808 07:00:54.881497 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.881505 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.881551 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.881660 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.881661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.881662 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:54.881724 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.881726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.882277 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.882278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.883022 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.883024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.883516 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.883517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.883591 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.883595 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.883635 (+ 40us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.883691 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260808 07:00:54.883795 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.881298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:45529 (local address 127.21.28.65:52249)
0808 07:00:54.881346 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.881356 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:54.881391 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:54.881524 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:54.881525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:54.881527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:54.881590 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.881592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.882221 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.882222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.883004 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:54.883006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.883565 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.883565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:54.883614 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.883617 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 07:00:54.883686 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 07:00:54.883722 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":32}
I20260808 07:00:54.884123 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb80467a06543ce84cbc734acad822c" candidate_uuid: "6e99a915b92c45dcb0b957ea0ff22421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8875135098ad4651aea91b765ebf2aa6" is_pre_election: true
I20260808 07:00:54.884267 18311 raft_consensus.cc:2454] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e99a915b92c45dcb0b957ea0ff22421 in term 0.
I20260808 07:00:54.884284 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb80467a06543ce84cbc734acad822c" candidate_uuid: "6e99a915b92c45dcb0b957ea0ff22421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2fcdb884a24d0db994610a7a01d8df" is_pre_election: true
I20260808 07:00:54.884411 18100 raft_consensus.cc:2454] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e99a915b92c45dcb0b957ea0ff22421 in term 0.
I20260808 07:00:54.884469 17783 leader_election.cc:302] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e99a915b92c45dcb0b957ea0ff22421, 8875135098ad4651aea91b765ebf2aa6; no voters:
I20260808 07:00:54.884579 18421 raft_consensus.cc:2781] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:00:54.884634 18421 raft_consensus.cc:482] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:00:54.884661 18421 raft_consensus.cc:3037] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:54.885267 18421 raft_consensus.cc:504] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.885377 18421 leader_election.cc:288] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [CANDIDATE]: Term 1 election: Requested vote from peers ff2fcdb884a24d0db994610a7a01d8df (127.21.28.66:45529), 8875135098ad4651aea91b765ebf2aa6 (127.21.28.67:39787)
I20260808 07:00:54.885517 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb80467a06543ce84cbc734acad822c" candidate_uuid: "6e99a915b92c45dcb0b957ea0ff22421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8875135098ad4651aea91b765ebf2aa6"
I20260808 07:00:54.885536 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb80467a06543ce84cbc734acad822c" candidate_uuid: "6e99a915b92c45dcb0b957ea0ff22421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2fcdb884a24d0db994610a7a01d8df"
I20260808 07:00:54.885586 18311 raft_consensus.cc:3037] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:54.885603 18100 raft_consensus.cc:3037] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:54.886234 18311 raft_consensus.cc:2454] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e99a915b92c45dcb0b957ea0ff22421 in term 1.
I20260808 07:00:54.886245 18100 raft_consensus.cc:2454] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e99a915b92c45dcb0b957ea0ff22421 in term 1.
I20260808 07:00:54.886382 17783 leader_election.cc:302] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e99a915b92c45dcb0b957ea0ff22421, 8875135098ad4651aea91b765ebf2aa6; no voters:
I20260808 07:00:54.886466 18421 raft_consensus.cc:2781] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:54.886584 18421 raft_consensus.cc:686] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [term 1 LEADER]: Becoming Leader. State: Replica: 6e99a915b92c45dcb0b957ea0ff22421, State: Running, Role: LEADER
I20260808 07:00:54.886659 18421 consensus_queue.cc:237] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } }
I20260808 07:00:54.887355 17647 catalog_manager.cc:5495] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e99a915b92c45dcb0b957ea0ff22421 (127.21.28.65). New cstate: current_term: 1 leader_uuid: "6e99a915b92c45dcb0b957ea0ff22421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e99a915b92c45dcb0b957ea0ff22421" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 43583 } health_report { overall_health: HEALTHY } } }
I20260808 07:00:54.893443 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.891539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.21.28.65:43583)
0808 07:00:54.891590 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:54.891592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:54.891598 (+ 6us) server_negotiation.cc:407] Connection header received
0808 07:00:54.891621 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:54.891621 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:54.891635 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:54.891733 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:00:54.891836 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.892247 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.892905 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:54.893065 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:54.893104 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:54.893202 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:00:54.893278 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:00:54.893348 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:54.893370 (+ 22us) server_negotiation.cc:299] Negotiation successful
0808 07:00:54.893377 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 07:00:54.894840 18311 raft_consensus.cc:1261] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 1 FOLLOWER]: Refusing update from remote peer 6e99a915b92c45dcb0b957ea0ff22421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:54.894840 18100 raft_consensus.cc:1261] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 1 FOLLOWER]: Refusing update from remote peer 6e99a915b92c45dcb0b957ea0ff22421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:54.895140 18424 consensus_queue.cc:1035] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8875135098ad4651aea91b765ebf2aa6" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:54.895251 18421 consensus_queue.cc:1035] T 0cb80467a06543ce84cbc734acad822c P 6e99a915b92c45dcb0b957ea0ff22421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2fcdb884a24d0db994610a7a01d8df" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:54.897635 18433 mvcc.cc:205] Tried to move back new op lower bound from 7316162375247233024 to 7316162375216291840. Current Snapshot: MvccSnapshot[applied={T|T < 7316162375247233024}]
I20260808 07:00:54.899477 17647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:54.899566 17647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:54.900238 17647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a4ef61061a34dd6b9f80d2d72599372: Sending DeleteTablet for 3 replicas of tablet 0cb80467a06543ce84cbc734acad822c
I20260808 07:00:54.900604 18065 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cb80467a06543ce84cbc734acad822c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:54 UTC) from {username='slave'} at 127.0.0.1:35026
I20260808 07:00:54.900650 18282 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cb80467a06543ce84cbc734acad822c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:54 UTC) from {username='slave'} at 127.0.0.1:57494
I20260808 07:00:54.900821 18451 tablet_replica.cc:330] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: stopping tablet replica
I20260808 07:00:54.900874 17858 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cb80467a06543ce84cbc734acad822c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:54 UTC) from {username='slave'} at 127.0.0.1:55994
I20260808 07:00:54.901101 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 17752
I20260808 07:00:54.912159 18452 tablet_replica.cc:330] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: stopping tablet replica
I20260808 07:00:54.912284 18452 raft_consensus.cc:2229] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:54.912379 18452 raft_consensus.cc:2258] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:54.913345 18452 ts_tablet_manager.cc:1905] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:54.915364 18452 ts_tablet_manager.cc:1918] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:54.915473 18452 log.cc:1196] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/0cb80467a06543ce84cbc734acad822c
I20260808 07:00:54.915844 18452 ts_tablet_manager.cc:1939] T 0cb80467a06543ce84cbc734acad822c P 8875135098ad4651aea91b765ebf2aa6: Deleting consensus metadata
I20260808 07:00:54.919907 17622 catalog_manager.cc:4841] TS 8875135098ad4651aea91b765ebf2aa6 (127.21.28.67:39787): tablet 0cb80467a06543ce84cbc734acad822c (table test-table [id=0f41706b5bb540f0b201470ad7734be1]) successfully deleted
I20260808 07:00:54.901118 18451 raft_consensus.cc:2229] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:54.921430 18451 raft_consensus.cc:2258] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:54.921973 18451 ts_tablet_manager.cc:1905] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:54.923456 18451 ts_tablet_manager.cc:1918] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:54.923530 18451 log.cc:1196] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/0cb80467a06543ce84cbc734acad822c
I20260808 07:00:54.923794 18451 ts_tablet_manager.cc:1939] T 0cb80467a06543ce84cbc734acad822c P ff2fcdb884a24d0db994610a7a01d8df: Deleting consensus metadata
I20260808 07:00:54.924221 17620 catalog_manager.cc:4841] TS ff2fcdb884a24d0db994610a7a01d8df (127.21.28.66:45529): tablet 0cb80467a06543ce84cbc734acad822c (table test-table [id=0f41706b5bb540f0b201470ad7734be1]) successfully deleted
W20260808 07:00:54.925737 17620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:43583 (error 108)
I20260808 07:00:54.925891 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 17973
I20260808 07:00:54.927322 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:54.926029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:43583 (local address 127.0.0.1:56018)
0808 07:00:54.927244 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:54.927290 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1192}
W20260808 07:00:54.927444 17620 catalog_manager.cc:4584] TS 6e99a915b92c45dcb0b957ea0ff22421 (127.21.28.65:43583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0cb80467a06543ce84cbc734acad822c: Network error: Client connection negotiation failed: client connection to 127.21.28.65:43583: connect: Connection refused (error 111)
I20260808 07:00:54.933804 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 18179
I20260808 07:00:54.941329 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 17592
2026-08-08T07:00:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (666 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:00:54 dist-test-slave-nj21 krb5kdc[18463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:00:54 dist-test-slave-nj21 krb5kdc[18463](info): set up 2 sockets
Aug 08 07:00:54 dist-test-slave-nj21 krb5kdc[18463](info): commencing operation
krb5kdc: starting...
W20260808 07:00:56.070981 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:00:56.179908 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:38803
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:38803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:56.257871 18484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:56.258059 18484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:56.258087 18484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:56.259202 18484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:56.259248 18484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:56.259265 18484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:56.259281 18484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:56.259315 18484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:56.261305 18484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:38803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:38803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:56.261601 18484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:56.261840 18484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:56.266470 18492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:56.266474 18494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:56.266531 18484 server_base.cc:1034] running on GCE node
W20260808 07:00:56.266467 18491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:56.266901 18484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:56.267206 18484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:56.268347 18484 hybrid_clock.cc:648] HybridClock initialized: now 1786172456268338 us; error 38 us; skew 500 ppm
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:56.270821 18484 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:00:56.271217 18484 webserver.cc:460] Webserver started at http://127.21.28.126:33225/ using document root <none> and password file <none>
I20260808 07:00:56.271447 18484 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:56.271492 18484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:56.271615 18484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:56.272662 18484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "86a166f7080a4d968e63640a6f60980b"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "3d760cdf070f34de1a2e0401f17923a6"
server_key_iv: "fbe43f840631502bc8347c604983892c"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.273001 18484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "86a166f7080a4d968e63640a6f60980b"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "3d760cdf070f34de1a2e0401f17923a6"
server_key_iv: "fbe43f840631502bc8347c604983892c"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.274288 18484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:56.274923 18502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.275058 18484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:56.275142 18484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "86a166f7080a4d968e63640a6f60980b"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "3d760cdf070f34de1a2e0401f17923a6"
server_key_iv: "fbe43f840631502bc8347c604983892c"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.275202 18484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:56.290486 18484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:56.292157 18484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:56.292289 18484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:56.296096 18484 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:38803
I20260808 07:00:56.296131 18585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:38803 every 8 connection(s)
I20260808 07:00:56.296442 18484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:56.297266 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:56.299464 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b: Bootstrap starting.
I20260808 07:00:56.300034 18586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:56.300258 18586 log.cc:824] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:56.300863 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b: No bootstrap required, opened a new log
I20260808 07:00:56.302359 18586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a166f7080a4d968e63640a6f60980b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38803 } }
I20260808 07:00:56.302497 18586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:56.302520 18586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a166f7080a4d968e63640a6f60980b, State: Initialized, Role: FOLLOWER
I20260808 07:00:56.302626 18586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a166f7080a4d968e63640a6f60980b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38803 } }
I20260808 07:00:56.302700 18586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:56.302731 18586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:56.302763 18586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:56.303318 18586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a166f7080a4d968e63640a6f60980b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38803 } }
I20260808 07:00:56.303413 18586 leader_election.cc:302] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a166f7080a4d968e63640a6f60980b; no voters:
I20260808 07:00:56.303543 18586 leader_election.cc:288] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:00:56.303596 18589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:56.303701 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 18484
I20260808 07:00:56.303736 18589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [term 1 LEADER]: Becoming Leader. State: Replica: 86a166f7080a4d968e63640a6f60980b, State: Running, Role: LEADER
I20260808 07:00:56.303809 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:00:56.303810 18586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:56.303809 18589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a166f7080a4d968e63640a6f60980b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38803 } }
I20260808 07:00:56.304036 21617 external_mini_cluster.cc:1428] Setting key 175c26f52d251ef430042e2bdb53098c
I20260808 07:00:56.304898 18590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a166f7080a4d968e63640a6f60980b. Latest consensus state: current_term: 1 leader_uuid: "86a166f7080a4d968e63640a6f60980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a166f7080a4d968e63640a6f60980b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38803 } } }
I20260808 07:00:56.304979 18590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:56.305183 18610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:56.305593 18591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a166f7080a4d968e63640a6f60980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a166f7080a4d968e63640a6f60980b" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38803 } } }
I20260808 07:00:56.305684 18591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:56.305795 18610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:56.307361 18610 catalog_manager.cc:1347] Generated new cluster ID: cd1ca569735e48ef8b36a37bb3e66e1a
I20260808 07:00:56.307415 18610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:56.314167 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.304716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.21.28.126:38803)
0808 07:00:56.304987 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.304996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.305030 (+ 34us) server_negotiation.cc:407] Connection header received
0808 07:00:56.305149 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.305157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.305224 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.305929 (+ 705us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:56.306133 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.307250 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.308067 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.308488 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.308581 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.310709 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:56.310713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:56.310713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:56.310725 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:56.312794 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.313106 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.313109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.313109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.313200 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.313368 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.313369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.313369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.313594 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:56.313733 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.313831 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.313973 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":73,"threads_started":1}
I20260808 07:00:56.328899 18610 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:00:56.329377 18610 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:56.334926 18610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b: Generated new TSK 0
I20260808 07:00:56.335103 18610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.28.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
Entry for principal kudu/127.21.28.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab.
I20260808 07:00:56.359637 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38803
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:56.437423 18620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:56.437604 18620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:56.437625 18620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:56.438719 18620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:56.438756 18620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:56.438784 18620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 07:00:56.440205 18620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.65.keytab
--principal=kudu/127.21.28.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:56.440423 18620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:56.440618 18620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:56.445138 18629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:56.445247 18620 server_base.cc:1034] running on GCE node
W20260808 07:00:56.445129 18630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:56.445127 18632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:56.445521 18620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:56.445706 18620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:56.446834 18620 hybrid_clock.cc:648] HybridClock initialized: now 1786172456446824 us; error 27 us; skew 500 ppm
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:56.449172 18620 init.cc:377] Logged in from keytab as kudu/127.21.28.65@KRBTEST.COM (short username kudu)
I20260808 07:00:56.449568 18620 webserver.cc:460] Webserver started at http://127.21.28.65:43473/ using document root <none> and password file <none>
I20260808 07:00:56.449748 18620 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:56.449790 18620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:56.449896 18620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:56.450824 18620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "6f3bf33de6944527bfee79adfd242e68"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "c77c79762832a6f5c5236d5179d373fb"
server_key_iv: "e37fcc383ba500968a590ead1d1d6014"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.451134 18620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "6f3bf33de6944527bfee79adfd242e68"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "c77c79762832a6f5c5236d5179d373fb"
server_key_iv: "e37fcc383ba500968a590ead1d1d6014"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.452360 18620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:00:56.452963 18645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.453105 18620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:56.453186 18620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "6f3bf33de6944527bfee79adfd242e68"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "c77c79762832a6f5c5236d5179d373fb"
server_key_iv: "e37fcc383ba500968a590ead1d1d6014"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.453243 18620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:56.471342 18620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:56.473282 18620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:56.473409 18620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:56.473599 18620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:56.473860 18620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:56.473894 18620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.473930 18620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:56.473955 18620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.479483 18620 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:33053
I20260808 07:00:56.479554 18825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:33053 every 8 connection(s)
I20260808 07:00:56.479836 18620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 07:00:56.483796 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 18620
I20260808 07:00:56.483881 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 07:00:56.484061 21617 external_mini_cluster.cc:1428] Setting key ed56535c02188cdfef09477b53f959d1
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.65@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:56.488131 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.480453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:40267 (local address 127.21.28.126:38803)
0808 07:00:56.480524 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.480527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.480673 (+ 146us) server_negotiation.cc:407] Connection header received
0808 07:00:56.481118 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.481119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.481129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.481173 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:56.482005 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.482490 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.483371 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.483552 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.483595 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.485682 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:56.485684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:56.485684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:56.485691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:56.486913 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.487262 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.487263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.487263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.487317 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.487535 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.487536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.487536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.487642 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:56.487721 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.487963 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.488043 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 07:00:56.488267 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.480437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38803 (local address 127.21.28.65:40267)
0808 07:00:56.480566 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:56.480598 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:56.481044 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:56.481229 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:56.481232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:56.481755 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:56.481933 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.481944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.482550 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.482551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.483314 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.483316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.483615 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.483615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.483742 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.484169 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:56.484179 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:56.485600 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:56.486986 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:56.486992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:56.486994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:56.487198 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:56.487370 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:56.487371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:56.487371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:56.487480 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:56.487710 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:56.487716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:56.487844 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 07:00:56.487927 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 07:00:56.488068 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260808 07:00:56.488739 18826 heartbeater.cc:347] Connected to a master server at 127.21.28.126:38803
I20260808 07:00:56.488811 18826 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:56.488966 18826 heartbeater.cc:511] Master 127.21.28.126:38803 requested a full tablet report, sending...
I20260808 07:00:56.489511 18528 ts_manager.cc:194] Registered new tserver with Master: 6f3bf33de6944527bfee79adfd242e68 (127.21.28.65:33053)
I20260808 07:00:56.490401 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.65@KRBTEST.COM'} at 127.21.28.65:40267
WARNING: no policy specified for kudu/127.21.28.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
Entry for principal kudu/127.21.28.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab.
I20260808 07:00:56.505911 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38803
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:56.584100 18832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:56.584275 18832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:56.584295 18832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:56.585337 18832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:56.585369 18832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:56.585399 18832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 07:00:56.586735 18832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.66.keytab
--principal=kudu/127.21.28.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:56.586899 18832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:56.587064 18832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:56.591657 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:56.591709 18841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:56.591729 18845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:56.591777 18832 server_base.cc:1034] running on GCE node
I20260808 07:00:56.592043 18832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:56.592245 18832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:56.593379 18832 hybrid_clock.cc:648] HybridClock initialized: now 1786172456593362 us; error 37 us; skew 500 ppm
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:56.595727 18832 init.cc:377] Logged in from keytab as kudu/127.21.28.66@KRBTEST.COM (short username kudu)
I20260808 07:00:56.596118 18832 webserver.cc:460] Webserver started at http://127.21.28.66:34381/ using document root <none> and password file <none>
I20260808 07:00:56.596299 18832 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:56.596336 18832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:56.596431 18832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:56.597404 18832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "5064c480c22e4735be59f00f702e8409"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "8dfa16c86c6247303f122220a52dc576"
server_key_iv: "c0a9d190022ad0e91652ea5709cfeecf"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.597674 18832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "5064c480c22e4735be59f00f702e8409"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "8dfa16c86c6247303f122220a52dc576"
server_key_iv: "c0a9d190022ad0e91652ea5709cfeecf"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.598788 18832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:56.599356 18857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.599517 18832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.599586 18832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "5064c480c22e4735be59f00f702e8409"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "8dfa16c86c6247303f122220a52dc576"
server_key_iv: "c0a9d190022ad0e91652ea5709cfeecf"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.599633 18832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:56.607451 18832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:56.609025 18832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:56.609124 18832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:56.609284 18832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:56.609570 18832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:56.609614 18832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.609647 18832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:56.609671 18832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.615231 18832 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:40399
I20260808 07:00:56.615286 19039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:40399 every 8 connection(s)
I20260808 07:00:56.615546 18832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 07:00:56.619833 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 18832
I20260808 07:00:56.619939 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 07:00:56.620102 21617 external_mini_cluster.cc:1428] Setting key a7d03ce246486d1a1538080a8f07ef5c
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.66@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:56.624334 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.616194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:52729 (local address 127.21.28.126:38803)
0808 07:00:56.616241 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.616243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.616417 (+ 174us) server_negotiation.cc:407] Connection header received
0808 07:00:56.616883 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.616884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.616927 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.616980 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:56.617860 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.618352 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.619294 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.619493 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.619547 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.621700 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:56.621702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:56.621702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:56.621709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:56.623122 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.623487 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.623488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.623489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.623548 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.623763 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.623764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.623765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.623890 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:56.623959 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.624167 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.624245 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 07:00:56.624388 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.616148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38803 (local address 127.21.28.66:52729)
0808 07:00:56.616307 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:56.616341 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:56.616787 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:56.617060 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:56.617064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:56.617591 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:56.617763 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.617768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.618433 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.618434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.619209 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.619211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.619558 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.619559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.619764 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.620212 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:56.620220 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:56.621621 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:56.623208 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:56.623214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:56.623216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:56.623422 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:56.623605 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:56.623605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:56.623605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:56.623720 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:56.623957 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:56.623960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:56.624062 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 07:00:56.624132 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 07:00:56.624242 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260808 07:00:56.624826 19040 heartbeater.cc:347] Connected to a master server at 127.21.28.126:38803
I20260808 07:00:56.624933 19040 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:56.625104 19040 heartbeater.cc:511] Master 127.21.28.126:38803 requested a full tablet report, sending...
I20260808 07:00:56.625429 18528 ts_manager.cc:194] Registered new tserver with Master: 5064c480c22e4735be59f00f702e8409 (127.21.28.66:40399)
I20260808 07:00:56.625770 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.66@KRBTEST.COM'} at 127.21.28.66:52729
WARNING: no policy specified for kudu/127.21.28.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
Entry for principal kudu/127.21.28.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab.
I20260808 07:00:56.643613 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38803
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:56.723019 19049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:56.723203 19049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:56.723223 19049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:56.724295 19049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:56.724326 19049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:56.724350 19049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 07:00:56.725697 19049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.67.keytab
--principal=kudu/127.21.28.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:38803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:56.725859 19049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:56.726023 19049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 07:00:56.730684 19049 server_base.cc:1034] running on GCE node
W20260808 07:00:56.730676 19056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:56.730638 19061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:56.730659 19055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:56.731060 19049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:56.731303 19049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:56.732424 19049 hybrid_clock.cc:648] HybridClock initialized: now 1786172456732413 us; error 30 us; skew 500 ppm
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:56.734764 19049 init.cc:377] Logged in from keytab as kudu/127.21.28.67@KRBTEST.COM (short username kudu)
I20260808 07:00:56.735173 19049 webserver.cc:460] Webserver started at http://127.21.28.67:42275/ using document root <none> and password file <none>
I20260808 07:00:56.735356 19049 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:56.735396 19049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:56.735510 19049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:56.736496 19049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "814c6e1c4d574862a17f8ef59dd624f2"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "ec32c61d7077ed5bf6f0565baf692528"
server_key_iv: "80ba938d4aceda44f2ebfdfb6be290eb"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.736809 19049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "814c6e1c4d574862a17f8ef59dd624f2"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "ec32c61d7077ed5bf6f0565baf692528"
server_key_iv: "80ba938d4aceda44f2ebfdfb6be290eb"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.738013 19049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:56.738606 19072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.738732 19049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:56.738813 19049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "814c6e1c4d574862a17f8ef59dd624f2"
format_stamp: "Formatted at 2026-08-08 07:00:56 on dist-test-slave-nj21"
server_key: "ec32c61d7077ed5bf6f0565baf692528"
server_key_iv: "80ba938d4aceda44f2ebfdfb6be290eb"
server_key_version: "encryptionkey@0"
I20260808 07:00:56.738869 19049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:56.747066 19049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:56.749033 19049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:56.749156 19049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:56.749336 19049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:56.749619 19049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:56.749651 19049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.749676 19049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:56.749701 19049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:56.755393 19049 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:36055
I20260808 07:00:56.755455 19261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:36055 every 8 connection(s)
I20260808 07:00:56.755749 19049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 07:00:56.757797 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 19049
I20260808 07:00:56.757881 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 07:00:56.758049 21617 external_mini_cluster.cc:1428] Setting key c618ec375a5dc771dcda7c7185430f02
Aug 08 07:00:56 dist-test-slave-nj21 krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172456, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.67@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:00:56.764052 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.756381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:55049 (local address 127.21.28.126:38803)
0808 07:00:56.756436 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.756438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.756621 (+ 183us) server_negotiation.cc:407] Connection header received
0808 07:00:56.757150 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.757152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.757163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.757216 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:56.758171 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.758810 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.759751 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.759965 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.760019 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.761800 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:56.761801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:56.761801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:56.761808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:56.762816 (+ 1008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.763156 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.763157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.763157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.763212 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.763412 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.763413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.763413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.763523 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:56.763579 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.763876 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.763962 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 07:00:56.764183 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.756361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38803 (local address 127.21.28.67:55049)
0808 07:00:56.756517 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:56.756549 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:56.757060 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:56.757276 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:56.757280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:56.757840 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:56.758061 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.758065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.758881 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.758882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.759659 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.759661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.760032 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.760033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.760158 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.760511 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:00:56.760518 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:56.761721 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:56.762885 (+ 1164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:56.762891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:56.762893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:56.763093 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:56.763265 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:00:56.763265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:00:56.763266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:00:56.763358 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:00:56.763577 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:56.763583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:56.763747 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 07:00:56.763846 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 07:00:56.763990 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260808 07:00:56.764586 19263 heartbeater.cc:347] Connected to a master server at 127.21.28.126:38803
I20260808 07:00:56.764669 19263 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:56.764827 19263 heartbeater.cc:511] Master 127.21.28.126:38803 requested a full tablet report, sending...
I20260808 07:00:56.765141 18528 ts_manager.cc:194] Registered new tserver with Master: 814c6e1c4d574862a17f8ef59dd624f2 (127.21.28.67:36055)
I20260808 07:00:56.765482 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.28.67@KRBTEST.COM'} at 127.21.28.67:55049
I20260808 07:00:56.769563 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:00:56.775772 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.770856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.21.28.126:38803)
0808 07:00:56.770901 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.770909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.770999 (+ 90us) server_negotiation.cc:407] Connection header received
0808 07:00:56.771041 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.771041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.771048 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.771089 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:56.771566 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.772050 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.772705 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.772867 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.772912 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.773806 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:56.773807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:56.773807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:56.773813 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:56.774959 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.775218 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.775218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.775218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.775258 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:56.775362 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:56.775362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:56.775362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:56.775479 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:56.775545 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.775621 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.775698 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 07:00:56.777036 18528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:00:56.778602 18528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:00:56.784298 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.780535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:36055 (local address 127.0.0.1:38396)
0808 07:00:56.780790 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:56.780830 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:56.780902 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:56.781080 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:56.781083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:56.781085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:56.781151 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.781153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.782105 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.782106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.783090 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.783093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.784001 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.784002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.784079 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.784093 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 07:00:56.784166 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 07:00:56.784217 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":60,"threads_started":1}
I20260808 07:00:56.784305 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.780606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.21.28.67:36055)
0808 07:00:56.780782 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.780786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.780913 (+ 127us) server_negotiation.cc:407] Connection header received
0808 07:00:56.780977 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.780979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.780994 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.781052 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:56.781219 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.782014 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.783174 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.783920 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.783995 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.784058 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.784182 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.784224 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260808 07:00:56.784646 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.780535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:33053 (local address 127.0.0.1:33916)
0808 07:00:56.780831 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:56.780839 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:56.780893 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:56.781349 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:56.781350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:56.781352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:56.781389 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.781391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.782638 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.782639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.783575 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.783577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.784479 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.784481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.784536 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.784538 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:56.784563 (+ 25us) client_negotiation.cc:240] Negotiation successful
0808 07:00:56.784589 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":4,"thread_start_us":50,"threads_started":1}
I20260808 07:00:56.784983 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.780698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.21.28.65:33053)
0808 07:00:56.781195 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.781200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.781209 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:56.781264 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.781266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.781281 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.781354 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:56.781475 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.782538 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.783646 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.784259 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.784801 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.784865 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.784888 (+ 23us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.784922 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":80,"threads_started":1}
I20260808 07:00:56.785086 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.780715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40399 (local address 127.0.0.1:41370)
0808 07:00:56.781463 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:56.781472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:56.781523 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:56.781778 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:56.781779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:56.781780 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:56.781823 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.781825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.782908 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.782909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.783949 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.783952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.784866 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.784867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.784927 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.784931 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:56.784983 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 07:00:56.785020 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260808 07:00:56.785233 19153 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae11cdc00f5473b8edeee9de9f4ace6 (DEFAULT_TABLE table=test-table [id=ffb914c5cabb4209924e4a1a079640a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:56.785367 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.780961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.21.28.66:40399)
0808 07:00:56.781150 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.781154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.781572 (+ 418us) server_negotiation.cc:407] Connection header received
0808 07:00:56.781621 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.781624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.781639 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.781702 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:56.781889 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.782828 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.784029 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.784714 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.784777 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.784833 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.785014 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.785290 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260808 07:00:56.785578 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae11cdc00f5473b8edeee9de9f4ace6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:56.785597 18729 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae11cdc00f5473b8edeee9de9f4ace6 (DEFAULT_TABLE table=test-table [id=ffb914c5cabb4209924e4a1a079640a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:56.785858 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae11cdc00f5473b8edeee9de9f4ace6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:56.785938 18935 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae11cdc00f5473b8edeee9de9f4ace6 (DEFAULT_TABLE table=test-table [id=ffb914c5cabb4209924e4a1a079640a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:56.786247 18935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae11cdc00f5473b8edeee9de9f4ace6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:56.788218 19288 tablet_bootstrap.cc:492] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: Bootstrap starting.
I20260808 07:00:56.788883 19288 tablet_bootstrap.cc:654] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:56.789014 19290 tablet_bootstrap.cc:492] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Bootstrap starting.
I20260808 07:00:56.789323 19288 log.cc:824] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:56.789516 19291 tablet_bootstrap.cc:492] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Bootstrap starting.
I20260808 07:00:56.789665 19290 tablet_bootstrap.cc:654] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:56.790073 19291 tablet_bootstrap.cc:654] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:56.790205 19290 log.cc:824] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:56.790274 19288 tablet_bootstrap.cc:492] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: No bootstrap required, opened a new log
I20260808 07:00:56.790346 19288 ts_tablet_manager.cc:1397] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:56.790639 19291 log.cc:824] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:56.790859 19290 tablet_bootstrap.cc:492] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: No bootstrap required, opened a new log
I20260808 07:00:56.790930 19290 ts_tablet_manager.cc:1397] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:56.791610 19291 tablet_bootstrap.cc:492] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: No bootstrap required, opened a new log
I20260808 07:00:56.791704 19291 ts_tablet_manager.cc:1397] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:56.792330 19288 raft_consensus.cc:348] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.792508 19288 raft_consensus.cc:374] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:56.792546 19288 raft_consensus.cc:729] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3bf33de6944527bfee79adfd242e68, State: Initialized, Role: FOLLOWER
I20260808 07:00:56.792481 19290 raft_consensus.cc:348] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.792636 19290 raft_consensus.cc:374] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:56.792675 19290 raft_consensus.cc:729] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814c6e1c4d574862a17f8ef59dd624f2, State: Initialized, Role: FOLLOWER
I20260808 07:00:56.792663 19288 consensus_queue.cc:260] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.792783 19290 consensus_queue.cc:260] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.792948 19288 ts_tablet_manager.cc:1428] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:56.792999 18826 heartbeater.cc:503] Master 127.21.28.126:38803 was elected leader, sending a full tablet report...
I20260808 07:00:56.793048 19290 ts_tablet_manager.cc:1428] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:56.793152 19263 heartbeater.cc:503] Master 127.21.28.126:38803 was elected leader, sending a full tablet report...
I20260808 07:00:56.793879 19291 raft_consensus.cc:348] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.794039 19291 raft_consensus.cc:374] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:56.794085 19291 raft_consensus.cc:729] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5064c480c22e4735be59f00f702e8409, State: Initialized, Role: FOLLOWER
I20260808 07:00:56.794189 19291 consensus_queue.cc:260] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.794435 19291 ts_tablet_manager.cc:1428] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 07:00:56.794440 19040 heartbeater.cc:503] Master 127.21.28.126:38803 was elected leader, sending a full tablet report...
W20260808 07:00:56.866070 19043 tablet.cc:2369] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:56.954717 19299 raft_consensus.cc:482] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:56.954854 19299 raft_consensus.cc:504] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.955224 19299 leader_election.cc:288] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f3bf33de6944527bfee79adfd242e68 (127.21.28.65:33053), 5064c480c22e4735be59f00f702e8409 (127.21.28.66:40399)
I20260808 07:00:56.959162 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.955471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:57399 (local address 127.21.28.65:33053)
0808 07:00:56.955531 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.955533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.955564 (+ 31us) server_negotiation.cc:407] Connection header received
0808 07:00:56.955656 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.955657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.955682 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.955847 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:56.956062 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.956835 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.958273 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.958832 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.958902 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.958911 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.959054 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.959092 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 07:00:56.959162 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.955728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:58921 (local address 127.21.28.66:40399)
0808 07:00:56.955793 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:56.955795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:56.955803 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:00:56.955862 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:56.955864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:56.955874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:56.955932 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:56.956073 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.956813 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.958272 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.958832 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.958908 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.958915 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:56.959054 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 07:00:56.959092 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 07:00:56.959195 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.955417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:33053 (local address 127.21.28.67:57399)
0808 07:00:56.955496 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:56.955509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:56.955584 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:56.955896 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:56.955897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:56.955898 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:56.955986 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.955989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.956900 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.956901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.958190 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.958194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.958911 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.958911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.958990 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.958992 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:56.959043 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 07:00:56.959087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":90}
I20260808 07:00:56.959200 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:56.955546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:40399 (local address 127.21.28.67:58921)
0808 07:00:56.955737 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:56.955745 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:56.955799 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:56.955955 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:56.955956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:56.955958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:56.956015 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.956016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.956879 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.956880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.958190 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:56.958194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:56.958904 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:56.958905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:56.958985 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:56.958990 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 07:00:56.959042 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 07:00:56.959087 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":56,"thread_start_us":98,"threads_started":1}
I20260808 07:00:56.959576 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae11cdc00f5473b8edeee9de9f4ace6" candidate_uuid: "814c6e1c4d574862a17f8ef59dd624f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3bf33de6944527bfee79adfd242e68" is_pre_election: true
I20260808 07:00:56.959757 18763 raft_consensus.cc:2454] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814c6e1c4d574862a17f8ef59dd624f2 in term 0.
I20260808 07:00:56.959789 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae11cdc00f5473b8edeee9de9f4ace6" candidate_uuid: "814c6e1c4d574862a17f8ef59dd624f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5064c480c22e4735be59f00f702e8409" is_pre_election: true
I20260808 07:00:56.959945 18972 raft_consensus.cc:2454] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814c6e1c4d574862a17f8ef59dd624f2 in term 0.
I20260808 07:00:56.959997 19075 leader_election.cc:302] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3bf33de6944527bfee79adfd242e68, 814c6e1c4d574862a17f8ef59dd624f2; no voters:
I20260808 07:00:56.960172 19299 raft_consensus.cc:2781] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:00:56.960211 19299 raft_consensus.cc:482] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:00:56.960235 19299 raft_consensus.cc:3037] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:56.960942 19299 raft_consensus.cc:504] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.961047 19299 leader_election.cc:288] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [CANDIDATE]: Term 1 election: Requested vote from peers 6f3bf33de6944527bfee79adfd242e68 (127.21.28.65:33053), 5064c480c22e4735be59f00f702e8409 (127.21.28.66:40399)
I20260808 07:00:56.961180 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae11cdc00f5473b8edeee9de9f4ace6" candidate_uuid: "814c6e1c4d574862a17f8ef59dd624f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3bf33de6944527bfee79adfd242e68"
I20260808 07:00:56.961226 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae11cdc00f5473b8edeee9de9f4ace6" candidate_uuid: "814c6e1c4d574862a17f8ef59dd624f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5064c480c22e4735be59f00f702e8409"
I20260808 07:00:56.961252 18763 raft_consensus.cc:3037] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:56.961300 18972 raft_consensus.cc:3037] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:56.962011 18763 raft_consensus.cc:2454] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 814c6e1c4d574862a17f8ef59dd624f2 in term 1.
I20260808 07:00:56.962021 18972 raft_consensus.cc:2454] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 814c6e1c4d574862a17f8ef59dd624f2 in term 1.
I20260808 07:00:56.962186 19075 leader_election.cc:302] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5064c480c22e4735be59f00f702e8409, 814c6e1c4d574862a17f8ef59dd624f2; no voters:
I20260808 07:00:56.962275 19299 raft_consensus.cc:2781] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:56.962400 19299 raft_consensus.cc:686] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 1 LEADER]: Becoming Leader. State: Replica: 814c6e1c4d574862a17f8ef59dd624f2, State: Running, Role: LEADER
I20260808 07:00:56.962477 19299 consensus_queue.cc:237] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } }
I20260808 07:00:56.963248 18528 catalog_manager.cc:5495] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 814c6e1c4d574862a17f8ef59dd624f2 (127.21.28.67). New cstate: current_term: 1 leader_uuid: "814c6e1c4d574862a17f8ef59dd624f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "814c6e1c4d574862a17f8ef59dd624f2" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 36055 } health_report { overall_health: HEALTHY } } }
W20260808 07:00:56.980473 18827 tablet.cc:2369] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:00:57.006302 19264 tablet.cc:2369] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:57.010321 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.008030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.21.28.67:36055)
0808 07:00:57.008092 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.008094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.008103 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:57.008139 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.008140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.008159 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.008218 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:00:57.008348 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.008816 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.009590 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.009788 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.009842 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.009995 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:00:57.010094 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:00:57.010162 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.010205 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.010227 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 07:00:57.012020 18763 raft_consensus.cc:1261] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 1 FOLLOWER]: Refusing update from remote peer 814c6e1c4d574862a17f8ef59dd624f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:57.012022 18972 raft_consensus.cc:1261] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 1 FOLLOWER]: Refusing update from remote peer 814c6e1c4d574862a17f8ef59dd624f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:57.012364 19299 consensus_queue.cc:1035] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3bf33de6944527bfee79adfd242e68" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 33053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:57.012509 19304 consensus_queue.cc:1035] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5064c480c22e4735be59f00f702e8409" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 40399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:57.015091 19311 mvcc.cc:205] Tried to move back new op lower bound from 7316162383918612480 to 7316162383718899712. Current Snapshot: MvccSnapshot[applied={T|T < 7316162383918612480}]
I20260808 07:00:57.015846 19312 mvcc.cc:205] Tried to move back new op lower bound from 7316162383918612480 to 7316162383718899712. Current Snapshot: MvccSnapshot[applied={T|T < 7316162383918612480 or (T in {7316162383918612480})}]
I20260808 07:00:57.016916 18528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:57.016996 18528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:57.017688 18528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86a166f7080a4d968e63640a6f60980b: Sending DeleteTablet for 3 replicas of tablet 7ae11cdc00f5473b8edeee9de9f4ace6
I20260808 07:00:57.018010 18935 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae11cdc00f5473b8edeee9de9f4ace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:57 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:41370
I20260808 07:00:57.018015 18729 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae11cdc00f5473b8edeee9de9f4ace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:57 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:33916
I20260808 07:00:57.018229 19322 tablet_replica.cc:330] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: stopping tablet replica
I20260808 07:00:57.018229 19321 tablet_replica.cc:330] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: stopping tablet replica
I20260808 07:00:57.018241 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae11cdc00f5473b8edeee9de9f4ace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:57 UTC) from {username='kudu', principal='kudu/127.21.28.126@KRBTEST.COM'} at 127.0.0.1:38396
I20260808 07:00:57.018329 19322 raft_consensus.cc:2229] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:57.018431 19322 raft_consensus.cc:2258] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:57.018436 19321 raft_consensus.cc:2229] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:57.018676 19321 raft_consensus.cc:2258] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:57.018748 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 18620
I20260808 07:00:57.018815 19322 ts_tablet_manager.cc:1905] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 6f3bf33de6944527bfee79adfd242e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:57.024849 19323 tablet_replica.cc:330] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: stopping tablet replica
I20260808 07:00:57.024952 19323 raft_consensus.cc:2229] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:00:57.025058 19323 raft_consensus.cc:2258] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:57.025380 19323 ts_tablet_manager.cc:1905] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:57.026777 19323 ts_tablet_manager.cc:1918] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:57.026839 19323 log.cc:1196] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/7ae11cdc00f5473b8edeee9de9f4ace6
I20260808 07:00:57.027081 19323 ts_tablet_manager.cc:1939] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 814c6e1c4d574862a17f8ef59dd624f2: Deleting consensus metadata
I20260808 07:00:57.027415 18506 catalog_manager.cc:4841] TS 814c6e1c4d574862a17f8ef59dd624f2 (127.21.28.67:36055): tablet 7ae11cdc00f5473b8edeee9de9f4ace6 (table test-table [id=ffb914c5cabb4209924e4a1a079640a5]) successfully deleted
I20260808 07:00:57.027946 19321 ts_tablet_manager.cc:1905] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:57.029157 19321 ts_tablet_manager.cc:1918] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:57.029217 19321 log.cc:1196] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786172218768454-21617-0/minicluster-data/ts-1/wal/wals/7ae11cdc00f5473b8edeee9de9f4ace6
I20260808 07:00:57.029464 19321 ts_tablet_manager.cc:1939] T 7ae11cdc00f5473b8edeee9de9f4ace6 P 5064c480c22e4735be59f00f702e8409: Deleting consensus metadata
I20260808 07:00:57.029805 18504 catalog_manager.cc:4841] TS 5064c480c22e4735be59f00f702e8409 (127.21.28.66:40399): tablet 7ae11cdc00f5473b8edeee9de9f4ace6 (table test-table [id=ffb914c5cabb4209924e4a1a079640a5]) successfully deleted
W20260808 07:00:57.031279 18504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:33053 (error 108)
I20260808 07:00:57.031565 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 18832
I20260808 07:00:57.031692 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.031562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:33053 (local address 127.0.0.1:33926)
0808 07:00:57.031604 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.031639 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:33053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 07:00:57.033182 18504 catalog_manager.cc:4584] TS 6f3bf33de6944527bfee79adfd242e68 (127.21.28.65:33053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ae11cdc00f5473b8edeee9de9f4ace6: Network error: Client connection negotiation failed: client connection to 127.21.28.65:33053: connect: Connection refused (error 111)
I20260808 07:00:57.038566 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 19049
I20260808 07:00:57.048534 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 18484
2026-08-08T07:00:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2107 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T07:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:00:57Z Disabled control of system clock
I20260808 07:00:57.078802 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:33531
--webserver_interface=127.21.28.126
--webserver_port=0
--builtin_ntp_servers=127.21.28.84:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:57.156477 19328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:57.156687 19328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:57.156713 19328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:57.157816 19328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:57.157855 19328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:57.157871 19328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:57.157887 19328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:57.157909 19328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:57.159198 19328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:33531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:57.159436 19328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:57.159716 19328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:57.164121 19343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:57.164180 19341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:57.164297 19328 server_base.cc:1034] running on GCE node
W20260808 07:00:57.164153 19336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:57.164822 19328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:57.165082 19328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:57.166214 19328 hybrid_clock.cc:648] HybridClock initialized: now 1786172457166204 us; error 38 us; skew 500 ppm
I20260808 07:00:57.167296 19328 webserver.cc:460] Webserver started at http://127.21.28.126:45731/ using document root <none> and password file <none>
I20260808 07:00:57.167521 19328 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:57.167570 19328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:57.167726 19328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:57.168689 19328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "8ed73c13a5a186be518bdbda83f8140c"
server_key_iv: "9f0c78646bbaac57bc607ffd47905676"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.169000 19328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "8ed73c13a5a186be518bdbda83f8140c"
server_key_iv: "9f0c78646bbaac57bc607ffd47905676"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.170132 19328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:57.170737 19351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.170887 19328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:57.170977 19328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "8ed73c13a5a186be518bdbda83f8140c"
server_key_iv: "9f0c78646bbaac57bc607ffd47905676"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.171033 19328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:57.185146 19328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:57.185387 19328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:57.185513 19328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:57.189093 19328 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:33531
I20260808 07:00:57.189124 19431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:33531 every 8 connection(s)
I20260808 07:00:57.189395 19328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:57.190267 19432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:57.192409 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 19328
I20260808 07:00:57.192433 19432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2: Bootstrap starting.
I20260808 07:00:57.192497 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:00:57.192652 21617 external_mini_cluster.cc:1428] Setting key a4fd16398f8bac947ba1f1f0a9d23e26
I20260808 07:00:57.193042 19432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:57.193377 19432 log.cc:824] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:57.194032 19432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2: No bootstrap required, opened a new log
I20260808 07:00:57.195989 19432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33531 } }
I20260808 07:00:57.196148 19432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:57.196184 19432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05106d1fb7cd4c2a8ab66cc49e5701a2, State: Initialized, Role: FOLLOWER
I20260808 07:00:57.196307 19432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33531 } }
I20260808 07:00:57.196370 19432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:57.196393 19432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:57.196435 19432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:57.197045 19432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33531 } }
I20260808 07:00:57.197147 19432 leader_election.cc:302] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05106d1fb7cd4c2a8ab66cc49e5701a2; no voters:
I20260808 07:00:57.197314 19432 leader_election.cc:288] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:00:57.197352 19444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:57.197482 19444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [term 1 LEADER]: Becoming Leader. State: Replica: 05106d1fb7cd4c2a8ab66cc49e5701a2, State: Running, Role: LEADER
I20260808 07:00:57.197572 19444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33531 } }
I20260808 07:00:57.197575 19432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:57.197956 19447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33531 } } }
I20260808 07:00:57.198031 19447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:57.198211 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.193078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.21.28.126:33531)
0808 07:00:57.193246 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.193252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.193276 (+ 24us) server_negotiation.cc:407] Connection header received
0808 07:00:57.193722 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.193728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.193782 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.193890 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:57.194127 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.194987 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.196105 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.196611 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.196695 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.197302 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:57.197305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:57.197306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:57.197506 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:57.197545 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:57.197559 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:57.197905 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.197936 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.198011 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260808 07:00:57.198208 19446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05106d1fb7cd4c2a8ab66cc49e5701a2. Latest consensus state: current_term: 1 leader_uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05106d1fb7cd4c2a8ab66cc49e5701a2" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33531 } } }
I20260808 07:00:57.198292 19446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:57.198861 19451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:57.199313 19451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:00:57.200654 19451 catalog_manager.cc:1347] Generated new cluster ID: 2ab00723cee24d5b9d85372030ebe8ae
I20260808 07:00:57.200706 19451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:00:57.210682 19451 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:00:57.211135 19451 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:57.213761 19451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2: Generated new TSK 0
I20260808 07:00:57.213910 19451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:00:57.218947 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.65:0
--local_ip_for_outbound_sockets=127.21.28.65
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33531
--builtin_ntp_servers=127.21.28.84:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:57.298363 19469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:57.298563 19469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:57.298584 19469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:57.299736 19469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:57.299770 19469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:57.299798 19469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.65
I20260808 07:00:57.301076 19469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.28.65
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.65
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:57.301266 19469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:57.301529 19469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:57.307204 19475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:57.307178 19478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:57.307178 19476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:57.307386 19469 server_base.cc:1034] running on GCE node
I20260808 07:00:57.307654 19469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:57.307943 19469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:57.309113 19469 hybrid_clock.cc:648] HybridClock initialized: now 1786172457309095 us; error 32 us; skew 500 ppm
I20260808 07:00:57.310441 19469 webserver.cc:460] Webserver started at http://127.21.28.65:34399/ using document root <none> and password file <none>
I20260808 07:00:57.310700 19469 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:57.310779 19469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:57.310923 19469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:57.312239 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/data/instance:
uuid: "41eddb673df84e119c3636de7344bf9c"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "515a47ab6d15018933a82714ebc35d51"
server_key_iv: "bf2fec5d3fa250ee681b061c3d94ec48"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.312635 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance:
uuid: "41eddb673df84e119c3636de7344bf9c"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "515a47ab6d15018933a82714ebc35d51"
server_key_iv: "bf2fec5d3fa250ee681b061c3d94ec48"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.314263 19469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 07:00:57.315045 19488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.315197 19469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:57.315289 19469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/wal
uuid: "41eddb673df84e119c3636de7344bf9c"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "515a47ab6d15018933a82714ebc35d51"
server_key_iv: "bf2fec5d3fa250ee681b061c3d94ec48"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.315351 19469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:57.329788 19469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:57.329995 19469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:57.330086 19469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:57.330255 19469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:57.330505 19469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:57.330534 19469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.330554 19469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:57.330569 19469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.335880 19469 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.65:35725
I20260808 07:00:57.335923 19664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.65:35725 every 8 connection(s)
I20260808 07:00:57.336200 19469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/data/info.pb
I20260808 07:00:57.340582 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.336825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.65:33595 (local address 127.21.28.126:33531)
0808 07:00:57.336898 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.336900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.337092 (+ 192us) server_negotiation.cc:407] Connection header received
0808 07:00:57.337512 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.337513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.337525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.337575 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:57.338140 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.338624 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.339482 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.339653 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.339735 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.340403 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:57.340404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:57.340404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:57.340408 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:57.340418 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:57.340421 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:57.340449 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.340509 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.340526 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 07:00:57.340766 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.336807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33531 (local address 127.21.28.65:33595)
0808 07:00:57.336983 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.337013 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:57.337441 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:57.337622 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:57.337624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:57.337632 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:57.338080 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.338085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.338681 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.338681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.339422 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.339424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.339735 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.339736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.339839 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.340297 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:57.340297 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:57.340322 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:57.340342 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:57.340459 (+ 117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:57.340461 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:57.340463 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:57.340505 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 07:00:57.340576 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260808 07:00:57.341184 19665 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33531
I20260808 07:00:57.341262 19665 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:57.341423 19665 heartbeater.cc:511] Master 127.21.28.126:33531 requested a full tablet report, sending...
I20260808 07:00:57.341848 19374 ts_manager.cc:194] Registered new tserver with Master: 41eddb673df84e119c3636de7344bf9c (127.21.28.65:35725)
I20260808 07:00:57.342423 19374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.65:33595
I20260808 07:00:57.343796 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 19469
I20260808 07:00:57.343892 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-0/wal/instance
I20260808 07:00:57.344065 21617 external_mini_cluster.cc:1428] Setting key 7b706d81473f2ba319820d3ec1e9777b
I20260808 07:00:57.344872 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.66:0
--local_ip_for_outbound_sockets=127.21.28.66
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33531
--builtin_ntp_servers=127.21.28.84:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 07:00:57.424049 19669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:57.424227 19669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:57.424247 19669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:57.425346 19669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:57.425386 19669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:57.425408 19669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.66
I20260808 07:00:57.426680 19669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.28.66
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.66
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:57.426887 19669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:57.427142 19669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:57.431694 19683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:57.431664 19688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:57.431691 19679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:57.431952 19669 server_base.cc:1034] running on GCE node
I20260808 07:00:57.432137 19669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:57.432344 19669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:57.433478 19669 hybrid_clock.cc:648] HybridClock initialized: now 1786172457433465 us; error 28 us; skew 500 ppm
I20260808 07:00:57.434648 19669 webserver.cc:460] Webserver started at http://127.21.28.66:42509/ using document root <none> and password file <none>
I20260808 07:00:57.434854 19669 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:57.434901 19669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:57.435048 19669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:57.436072 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/data/instance:
uuid: "b6249ef37a424434a5a7c9555248045b"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "953f22625a328f13e7a463380a951f3f"
server_key_iv: "553a1b5434d368c1674a85f5b23e0cb0"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.436378 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance:
uuid: "b6249ef37a424434a5a7c9555248045b"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "953f22625a328f13e7a463380a951f3f"
server_key_iv: "553a1b5434d368c1674a85f5b23e0cb0"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.437531 19669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:00:57.438117 19695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.438277 19669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:57.438357 19669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/wal
uuid: "b6249ef37a424434a5a7c9555248045b"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "953f22625a328f13e7a463380a951f3f"
server_key_iv: "553a1b5434d368c1674a85f5b23e0cb0"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.438416 19669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:57.458093 19669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:57.458329 19669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:57.458442 19669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:57.458631 19669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:57.458921 19669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:57.458954 19669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.458981 19669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:57.459005 19669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.464586 19669 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.66:44313
I20260808 07:00:57.464622 19878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.66:44313 every 8 connection(s)
I20260808 07:00:57.464890 19669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/data/info.pb
I20260808 07:00:57.469547 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.465506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:53147 (local address 127.21.28.126:33531)
0808 07:00:57.465560 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.465562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.465748 (+ 186us) server_negotiation.cc:407] Connection header received
0808 07:00:57.466225 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.466226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.466239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.466291 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:57.466905 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.467408 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.468367 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.468565 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.468620 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.469280 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:57.469282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:57.469282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:57.469287 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:57.469303 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:57.469308 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:57.469349 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.469428 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.469466 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 07:00:57.469717 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.465483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33531 (local address 127.21.28.66:53147)
0808 07:00:57.465641 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.465671 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:57.466127 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:57.466353 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:57.466356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:57.466364 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:57.466815 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.466819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.467488 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.467489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.468281 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.468283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.468633 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.468634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.468750 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.469154 (+ 404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:57.469155 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:57.469179 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:57.469200 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:57.469355 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:57.469360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:57.469362 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 07:00:57.469422 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 07:00:57.469528 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260808 07:00:57.469894 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 19669
I20260808 07:00:57.469998 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-1/wal/instance
I20260808 07:00:57.470149 19879 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33531
I20260808 07:00:57.470196 21617 external_mini_cluster.cc:1428] Setting key bf1508487018a539cd8e491220bf3515
I20260808 07:00:57.470238 19879 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:57.470429 19879 heartbeater.cc:511] Master 127.21.28.126:33531 requested a full tablet report, sending...
I20260808 07:00:57.470791 19374 ts_manager.cc:194] Registered new tserver with Master: b6249ef37a424434a5a7c9555248045b (127.21.28.66:44313)
I20260808 07:00:57.471045 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.28.67:0
--local_ip_for_outbound_sockets=127.21.28.67
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33531
--builtin_ntp_servers=127.21.28.84:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 07:00:57.471240 19374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.66:53147
W20260808 07:00:57.551347 19884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:57.551537 19884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:57.551565 19884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:57.552680 19884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:57.552721 19884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:57.552759 19884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.28.67
I20260808 07:00:57.554065 19884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.28.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.28.67
--webserver_port=0
--tserver_master_addrs=127.21.28.126:33531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.28.67
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:57.554284 19884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:57.554546 19884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 07:00:57.559170 19884 server_base.cc:1034] running on GCE node
W20260808 07:00:57.559160 19892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:57.559144 19895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:57.559155 19893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:57.559588 19884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:57.559818 19884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:57.560971 19884 hybrid_clock.cc:648] HybridClock initialized: now 1786172457560955 us; error 29 us; skew 500 ppm
I20260808 07:00:57.562085 19884 webserver.cc:460] Webserver started at http://127.21.28.67:43941/ using document root <none> and password file <none>
I20260808 07:00:57.562279 19884 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:57.562325 19884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:57.562441 19884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:57.563388 19884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/data/instance:
uuid: "c8a3af35ea564424977b54bdac762e9a"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "0a60fac9823a02c57e7155a8c4f8b480"
server_key_iv: "da45d0fd34226abb53b4903d3e754d79"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.563740 19884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance:
uuid: "c8a3af35ea564424977b54bdac762e9a"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "0a60fac9823a02c57e7155a8c4f8b480"
server_key_iv: "da45d0fd34226abb53b4903d3e754d79"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.564908 19884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:00:57.565510 19902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.565661 19884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:57.565739 19884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/wal
uuid: "c8a3af35ea564424977b54bdac762e9a"
format_stamp: "Formatted at 2026-08-08 07:00:57 on dist-test-slave-nj21"
server_key: "0a60fac9823a02c57e7155a8c4f8b480"
server_key_iv: "da45d0fd34226abb53b4903d3e754d79"
server_key_version: "encryptionkey@0"
I20260808 07:00:57.565802 19884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:57.588518 19884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:57.588771 19884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:57.588891 19884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:57.589098 19884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 07:00:57.589406 19884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 07:00:57.589452 19884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.589483 19884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 07:00:57.589507 19884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:57.594930 19884 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.67:43047
I20260808 07:00:57.594985 20089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.67:43047 every 8 connection(s)
I20260808 07:00:57.595264 19884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/data/info.pb
I20260808 07:00:57.595850 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 19884
I20260808 07:00:57.595956 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/wal/instance
I20260808 07:00:57.596171 21617 external_mini_cluster.cc:1428] Setting key 204ad0e3a81028ef545b7f82eed29eaa
I20260808 07:00:57.600111 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.595946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.67:41489 (local address 127.21.28.126:33531)
0808 07:00:57.596000 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.596001 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.596272 (+ 271us) server_negotiation.cc:407] Connection header received
0808 07:00:57.596745 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.596746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.596759 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.596809 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:57.597453 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.597982 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.598969 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.599165 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.599210 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.599913 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:57.599914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:57.599914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:57.599918 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:57.599930 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:57.599932 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:57.599968 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.600015 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.600033 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 07:00:57.600306 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.595952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:33531 (local address 127.21.28.67:41489)
0808 07:00:57.596142 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.596175 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:57.596672 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:57.596856 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:57.596859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:57.596866 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:00:57.597378 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.597382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.598052 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.598053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.598884 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.598887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.599223 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.599224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.599327 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.599799 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 07:00:57.599800 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:00:57.599825 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 07:00:57.599847 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:00:57.599970 (+ 123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:00:57.599972 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:00:57.599975 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 07:00:57.600015 (+ 40us) client_negotiation.cc:240] Negotiation successful
0808 07:00:57.600109 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260808 07:00:57.600705 20090 heartbeater.cc:347] Connected to a master server at 127.21.28.126:33531
I20260808 07:00:57.600775 20090 heartbeater.cc:464] Registering TS with master...
I20260808 07:00:57.600927 20090 heartbeater.cc:511] Master 127.21.28.126:33531 requested a full tablet report, sending...
I20260808 07:00:57.601240 19374 ts_manager.cc:194] Registered new tserver with Master: c8a3af35ea564424977b54bdac762e9a (127.21.28.67:43047)
I20260808 07:00:57.601584 19374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.28.67:41489
I20260808 07:00:57.607626 21617 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 07:00:57.611296 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.608993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.21.28.126:33531)
0808 07:00:57.609042 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.609044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.609139 (+ 95us) server_negotiation.cc:407] Connection header received
0808 07:00:57.609178 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.609179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.609187 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.609231 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:57.609354 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.609798 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.610467 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.610632 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.610671 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.611138 (+ 467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:57.611139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:57.611139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:00:57.611143 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:57.611153 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:00:57.611156 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:57.611194 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.611230 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.611247 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 07:00:57.612566 19374 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 07:00:57.614048 19374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 07:00:57.619879 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.616431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.21.28.65:35725)
0808 07:00:57.616632 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.616636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.616645 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:57.616678 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.616679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.616690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.616930 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:57.616995 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.617675 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.618681 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.619442 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.619515 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.619602 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.619759 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.619799 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
I20260808 07:00:57.620123 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.616236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.21.28.66:44313)
0808 07:00:57.616393 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.616396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.616406 (+ 10us) server_negotiation.cc:407] Connection header received
0808 07:00:57.616444 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.616446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.616461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.616545 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:57.616807 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.617874 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.619061 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.619583 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.619915 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.620005 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.620029 (+ 24us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.620066 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260808 07:00:57.620134 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.616069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.21.28.67:43047)
0808 07:00:57.616234 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.616238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.616246 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:00:57.616270 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.616271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.616281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.616343 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:57.616483 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.617513 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.618849 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.619546 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.619599 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.619665 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.620028 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.620063 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260808 07:00:57.620261 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.615841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:43047 (local address 127.0.0.1:53664)
0808 07:00:57.616020 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.616055 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:57.616116 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:57.616354 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:57.616357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:57.616359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:57.616424 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.616425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.617596 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.617598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.618782 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.618785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.619721 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.619722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.619783 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.619787 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:57.619835 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 07:00:57.619873 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260808 07:00:57.620344 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.615842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:35725 (local address 127.0.0.1:33740)
0808 07:00:57.616086 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.616094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:57.616134 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:57.616782 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:57.616784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:57.616786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:57.616831 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.616832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.617754 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.617756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.618595 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.618598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.619527 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.619528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.619600 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.619615 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 07:00:57.619743 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 07:00:57.619821 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":52,"spinlock_wait_cycles":384,"thread_start_us":105,"threads_started":1}
I20260808 07:00:57.620354 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.615842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:44313 (local address 127.0.0.1:41944)
0808 07:00:57.616023 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.616057 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:57.616117 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:57.616539 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:57.616540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:57.616542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:57.616730 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.616732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.617947 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.617948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.618994 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.618996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.619698 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.619699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.619772 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.619776 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:57.619804 (+ 28us) client_negotiation.cc:240] Negotiation successful
0808 07:00:57.619835 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260808 07:00:57.620947 19986 tablet_service.cc:1467] Processing CreateTablet for tablet bcc5e6cc42b741baae4776143901fff8 (DEFAULT_TABLE table=test-table [id=ee7ed4d6734d48fc88c00da19c230b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:57.620947 19561 tablet_service.cc:1467] Processing CreateTablet for tablet bcc5e6cc42b741baae4776143901fff8 (DEFAULT_TABLE table=test-table [id=ee7ed4d6734d48fc88c00da19c230b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:57.621176 19778 tablet_service.cc:1467] Processing CreateTablet for tablet bcc5e6cc42b741baae4776143901fff8 (DEFAULT_TABLE table=test-table [id=ee7ed4d6734d48fc88c00da19c230b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 07:00:57.621296 19561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc5e6cc42b741baae4776143901fff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:57.621296 19986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc5e6cc42b741baae4776143901fff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:57.621433 19778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc5e6cc42b741baae4776143901fff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:57.624028 20108 tablet_bootstrap.cc:492] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: Bootstrap starting.
I20260808 07:00:57.624492 20110 tablet_bootstrap.cc:492] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c: Bootstrap starting.
I20260808 07:00:57.624572 20108 tablet_bootstrap.cc:654] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:57.624884 20108 log.cc:824] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:57.625106 20110 tablet_bootstrap.cc:654] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:57.625409 20110 log.cc:824] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:57.625638 20108 tablet_bootstrap.cc:492] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: No bootstrap required, opened a new log
I20260808 07:00:57.625706 20108 ts_tablet_manager.cc:1397] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:57.625717 20109 tablet_bootstrap.cc:492] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Bootstrap starting.
I20260808 07:00:57.626024 20110 tablet_bootstrap.cc:492] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c: No bootstrap required, opened a new log
I20260808 07:00:57.626083 20110 ts_tablet_manager.cc:1397] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 07:00:57.626327 20109 tablet_bootstrap.cc:654] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:57.626649 20109 log.cc:824] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:57.627616 20109 tablet_bootstrap.cc:492] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: No bootstrap required, opened a new log
I20260808 07:00:57.627707 20109 ts_tablet_manager.cc:1397] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 07:00:57.628096 20110 raft_consensus.cc:348] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.628221 20110 raft_consensus.cc:374] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:57.628253 20110 raft_consensus.cc:729] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41eddb673df84e119c3636de7344bf9c, State: Initialized, Role: FOLLOWER
I20260808 07:00:57.628172 20108 raft_consensus.cc:348] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.628321 20108 raft_consensus.cc:374] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:57.628362 20108 raft_consensus.cc:729] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6249ef37a424434a5a7c9555248045b, State: Initialized, Role: FOLLOWER
I20260808 07:00:57.628484 20108 consensus_queue.cc:260] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.628576 20110 consensus_queue.cc:260] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.628728 20108 ts_tablet_manager.cc:1428] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:57.628841 20110 ts_tablet_manager.cc:1428] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:57.628844 19879 heartbeater.cc:503] Master 127.21.28.126:33531 was elected leader, sending a full tablet report...
I20260808 07:00:57.628958 19665 heartbeater.cc:503] Master 127.21.28.126:33531 was elected leader, sending a full tablet report...
I20260808 07:00:57.630194 20109 raft_consensus.cc:348] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.630311 20109 raft_consensus.cc:374] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:57.630330 20109 raft_consensus.cc:729] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a3af35ea564424977b54bdac762e9a, State: Initialized, Role: FOLLOWER
I20260808 07:00:57.630438 20109 consensus_queue.cc:260] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.630645 20109 ts_tablet_manager.cc:1428] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 07:00:57.630690 20090 heartbeater.cc:503] Master 127.21.28.126:33531 was elected leader, sending a full tablet report...
W20260808 07:00:57.715407 19882 tablet.cc:2369] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:00:57.836839 19666 tablet.cc:2369] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 07:00:57.845752 20091 tablet.cc:2369] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 07:00:57.850749 20120 raft_consensus.cc:482] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 07:00:57.850847 20120 raft_consensus.cc:504] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.851155 20120 leader_election.cc:288] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a3af35ea564424977b54bdac762e9a (127.21.28.67:43047), 41eddb673df84e119c3636de7344bf9c (127.21.28.65:35725)
I20260808 07:00:57.854741 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.851383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:54413 (local address 127.21.28.67:43047)
0808 07:00:57.851444 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.851447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.851455 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:00:57.851486 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.851487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.851496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.851556 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:57.851739 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.852480 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.853870 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.854398 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.854469 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.854478 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.854636 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.854674 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 07:00:57.854741 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.851405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.28.66:54921 (local address 127.21.28.65:35725)
0808 07:00:57.851462 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.851464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.851554 (+ 90us) server_negotiation.cc:407] Connection header received
0808 07:00:57.851635 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.851636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.851646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.851724 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 07:00:57.851859 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.852534 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.853869 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.854398 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.854469 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.854478 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.854636 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.854674 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 07:00:57.854770 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.851302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.67:43047 (local address 127.21.28.66:54413)
0808 07:00:57.851358 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.851369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:57.851410 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:57.851571 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:57.851572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:57.851574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:57.851649 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.851651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.852546 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.852547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.853791 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.853794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.854474 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.854476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.854570 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.854574 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:57.854624 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 07:00:57.854685 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":26}
I20260808 07:00:57.854776 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.851322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:35725 (local address 127.21.28.66:54921)
0808 07:00:57.851507 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.851515 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:00:57.851577 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:00:57.851758 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:00:57.851759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:00:57.851760 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 07:00:57.851805 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.851806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.852604 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.852605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.853790 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:00:57.853794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.854476 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.854477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:00:57.854570 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.854574 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 07:00:57.854624 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 07:00:57.854686 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":50,"thread_start_us":103,"threads_started":1}
I20260808 07:00:57.855126 20014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc5e6cc42b741baae4776143901fff8" candidate_uuid: "b6249ef37a424434a5a7c9555248045b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a3af35ea564424977b54bdac762e9a" is_pre_election: true
I20260808 07:00:57.855150 19590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc5e6cc42b741baae4776143901fff8" candidate_uuid: "b6249ef37a424434a5a7c9555248045b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41eddb673df84e119c3636de7344bf9c" is_pre_election: true
I20260808 07:00:57.855257 19590 raft_consensus.cc:2454] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6249ef37a424434a5a7c9555248045b in term 0.
I20260808 07:00:57.855266 20014 raft_consensus.cc:2454] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6249ef37a424434a5a7c9555248045b in term 0.
I20260808 07:00:57.855491 19697 leader_election.cc:302] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41eddb673df84e119c3636de7344bf9c, b6249ef37a424434a5a7c9555248045b; no voters:
I20260808 07:00:57.855608 20120 raft_consensus.cc:2781] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 07:00:57.855657 20120 raft_consensus.cc:482] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 07:00:57.855705 20120 raft_consensus.cc:3037] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:57.856383 20120 raft_consensus.cc:504] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.856490 20120 leader_election.cc:288] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [CANDIDATE]: Term 1 election: Requested vote from peers c8a3af35ea564424977b54bdac762e9a (127.21.28.67:43047), 41eddb673df84e119c3636de7344bf9c (127.21.28.65:35725)
I20260808 07:00:57.856634 19590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc5e6cc42b741baae4776143901fff8" candidate_uuid: "b6249ef37a424434a5a7c9555248045b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41eddb673df84e119c3636de7344bf9c"
I20260808 07:00:57.856638 20014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc5e6cc42b741baae4776143901fff8" candidate_uuid: "b6249ef37a424434a5a7c9555248045b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a3af35ea564424977b54bdac762e9a"
I20260808 07:00:57.856709 19590 raft_consensus.cc:3037] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:57.856707 20014 raft_consensus.cc:3037] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:57.857371 20014 raft_consensus.cc:2454] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6249ef37a424434a5a7c9555248045b in term 1.
I20260808 07:00:57.857448 19590 raft_consensus.cc:2454] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6249ef37a424434a5a7c9555248045b in term 1.
I20260808 07:00:57.857542 19696 leader_election.cc:302] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6249ef37a424434a5a7c9555248045b, c8a3af35ea564424977b54bdac762e9a; no voters:
I20260808 07:00:57.857609 20120 raft_consensus.cc:2781] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:57.857748 20120 raft_consensus.cc:686] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 1 LEADER]: Becoming Leader. State: Replica: b6249ef37a424434a5a7c9555248045b, State: Running, Role: LEADER
I20260808 07:00:57.857821 20120 consensus_queue.cc:237] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } }
I20260808 07:00:57.858589 19373 catalog_manager.cc:5495] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b reported cstate change: term changed from 0 to 1, leader changed from <none> to b6249ef37a424434a5a7c9555248045b (127.21.28.66). New cstate: current_term: 1 leader_uuid: "b6249ef37a424434a5a7c9555248045b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6249ef37a424434a5a7c9555248045b" member_type: VOTER last_known_addr { host: "127.21.28.66" port: 44313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 } health_report { overall_health: UNKNOWN } } }
I20260808 07:00:57.899173 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.897038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.21.28.66:44313)
0808 07:00:57.897078 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:57.897080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:57.897089 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:00:57.897125 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:57.897125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:57.897141 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:57.897217 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:00:57.897321 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.897797 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.898526 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:57.898739 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:57.898789 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:57.898905 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:00:57.899015 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:00:57.899069 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:57.899082 (+ 13us) server_negotiation.cc:299] Negotiation successful
0808 07:00:57.899096 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 07:00:57.900875 20014 raft_consensus.cc:1261] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 1 FOLLOWER]: Refusing update from remote peer b6249ef37a424434a5a7c9555248045b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:57.900889 19590 raft_consensus.cc:1261] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [term 1 FOLLOWER]: Refusing update from remote peer b6249ef37a424434a5a7c9555248045b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:00:57.901191 20120 consensus_queue.cc:1035] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [LEADER]: Connected to new peer: Peer: permanent_uuid: "41eddb673df84e119c3636de7344bf9c" member_type: VOTER last_known_addr { host: "127.21.28.65" port: 35725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:57.901327 20127 consensus_queue.cc:1035] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a3af35ea564424977b54bdac762e9a" member_type: VOTER last_known_addr { host: "127.21.28.67" port: 43047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:00:57.903580 20134 mvcc.cc:205] Tried to move back new op lower bound from 7316162387559477248 to 7316162387386101760. Current Snapshot: MvccSnapshot[applied={T|T < 7316162387559477248}]
I20260808 07:00:57.903858 20136 mvcc.cc:205] Tried to move back new op lower bound from 7316162387559477248 to 7316162387386101760. Current Snapshot: MvccSnapshot[applied={T|T < 7316162387559477248}]
I20260808 07:00:57.905650 19373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:57.905730 19373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 07:00:57.906379 19373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05106d1fb7cd4c2a8ab66cc49e5701a2: Sending DeleteTablet for 3 replicas of tablet bcc5e6cc42b741baae4776143901fff8
I20260808 07:00:57.906682 19986 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc5e6cc42b741baae4776143901fff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:57 UTC) from {username='slave'} at 127.0.0.1:53664
I20260808 07:00:57.906706 19561 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc5e6cc42b741baae4776143901fff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:57 UTC) from {username='slave'} at 127.0.0.1:33740
I20260808 07:00:57.906912 20149 tablet_replica.cc:330] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: stopping tablet replica
I20260808 07:00:57.907004 20149 raft_consensus.cc:2229] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:57.906980 19778 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc5e6cc42b741baae4776143901fff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 07:00:57 UTC) from {username='slave'} at 127.0.0.1:41944
I20260808 07:00:57.907156 20148 tablet_replica.cc:330] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c: stopping tablet replica
I20260808 07:00:57.907243 20148 raft_consensus.cc:2229] T bcc5e6cc42b741baae4776143901fff8 P 41eddb673df84e119c3636de7344bf9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 07:00:57.907239 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 19469
I20260808 07:00:57.925625 20149 raft_consensus.cc:2258] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:57.926246 20149 ts_tablet_manager.cc:1905] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:57.927621 20150 tablet_replica.cc:330] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: stopping tablet replica
I20260808 07:00:57.927765 20150 raft_consensus.cc:2229] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 1 LEADER]: Raft consensus shutting down.
I20260808 07:00:57.927809 20149 ts_tablet_manager.cc:1918] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 07:00:57.927888 20150 raft_consensus.cc:2258] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 07:00:57.927888 20149 log.cc:1196] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Deleting WAL directory at /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786172218768454-21617-0/minicluster-data/ts-2/wal/wals/bcc5e6cc42b741baae4776143901fff8
I20260808 07:00:57.928159 20149 ts_tablet_manager.cc:1939] T bcc5e6cc42b741baae4776143901fff8 P c8a3af35ea564424977b54bdac762e9a: Deleting consensus metadata
I20260808 07:00:57.928315 20150 ts_tablet_manager.cc:1905] T bcc5e6cc42b741baae4776143901fff8 P b6249ef37a424434a5a7c9555248045b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 07:00:57.928606 19352 catalog_manager.cc:4841] TS c8a3af35ea564424977b54bdac762e9a (127.21.28.67:43047): tablet bcc5e6cc42b741baae4776143901fff8 (table test-table [id=ee7ed4d6734d48fc88c00da19c230b2d]) successfully deleted
W20260808 07:00:57.929425 19353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.28.65:35725 (error 108)
I20260808 07:00:57.929656 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 19669
I20260808 07:00:57.932086 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.929785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.65:35725 (local address 127.0.0.1:33754)
0808 07:00:57.932002 (+ 2217us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.932053 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.65:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2188}
W20260808 07:00:57.932247 19353 catalog_manager.cc:4584] TS 41eddb673df84e119c3636de7344bf9c (127.21.28.65:35725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcc5e6cc42b741baae4776143901fff8: Network error: Client connection negotiation failed: client connection to 127.21.28.65:35725: connect: Connection refused (error 111)
I20260808 07:00:57.941885 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:57.941772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.66:44313 (local address 127.0.0.1:41960)
0808 07:00:57.941815 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 07:00:57.941847 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.66:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 07:00:57.941892 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 19884
I20260808 07:00:57.948352 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 19328
2026-08-08T07:00:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (899 ms)
[----------] 6 tests from EncryptionPolicyTest (9005 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:00:57 dist-test-slave-nj21 krb5kdc[20156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:00:58 dist-test-slave-nj21 krb5kdc[20156](info): set up 2 sockets
Aug 08 07:00:58 dist-test-slave-nj21 krb5kdc[20156](info): commencing operation
krb5kdc: starting...
W20260808 07:00:59.070550 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:00:59 dist-test-slave-nj21 krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:00:59.180492 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:00:59.258579 20175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:00:59.258764 20175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:00:59.258795 20175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:00:59.259950 20175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:00:59.259989 20175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:00:59.260006 20175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:00:59.260023 20175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:00:59.260038 20175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:00:59.261499 20175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:00:59.261722 20175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:00:59.261919 20175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:00:59.266424 20184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:59.266579 20175 server_base.cc:1034] running on GCE node
W20260808 07:00:59.266450 20181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:00:59.266420 20182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:00:59.266891 20175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:00:59.267172 20175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:00:59.268312 20175 hybrid_clock.cc:648] HybridClock initialized: now 1786172459268300 us; error 26 us; skew 500 ppm
Aug 08 07:00:59 dist-test-slave-nj21 krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:00:59.271210 20175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:00:59.271811 20175 webserver.cc:460] Webserver started at http://127.0.0.1:41889/ using document root <none> and password file <none>
I20260808 07:00:59.272037 20175 fs_manager.cc:359] Metadata directory not provided
I20260808 07:00:59.272080 20175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:00:59.272198 20175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:00:59.273154 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "683525f4377a4694afe95f78e49d9562"
format_stamp: "Formatted at 2026-08-08 07:00:59 on dist-test-slave-nj21"
server_key: "35df19df9cbcfcfdd18b33915295b4f5"
server_key_iv: "e1d3faf436b3cfc129394ce4c9405844"
server_key_version: "encryptionkey@0"
I20260808 07:00:59.273468 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "683525f4377a4694afe95f78e49d9562"
format_stamp: "Formatted at 2026-08-08 07:00:59 on dist-test-slave-nj21"
server_key: "35df19df9cbcfcfdd18b33915295b4f5"
server_key_iv: "e1d3faf436b3cfc129394ce4c9405844"
server_key_version: "encryptionkey@0"
I20260808 07:00:59.274751 20175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:00:59.275388 20192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:00:59.275517 20175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:00:59.275601 20175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "683525f4377a4694afe95f78e49d9562"
format_stamp: "Formatted at 2026-08-08 07:00:59 on dist-test-slave-nj21"
server_key: "35df19df9cbcfcfdd18b33915295b4f5"
server_key_iv: "e1d3faf436b3cfc129394ce4c9405844"
server_key_version: "encryptionkey@0"
I20260808 07:00:59.275666 20175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:00:59.286226 20175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:00:59.287848 20175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:00:59.287961 20175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:00:59.291607 20175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42063
I20260808 07:00:59.291640 20271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42063 every 8 connection(s)
I20260808 07:00:59.291954 20175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:00:59.292721 20272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:00:59.294430 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 20175
I20260808 07:00:59.294528 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:00:59.294692 21617 external_mini_cluster.cc:1428] Setting key 1ff533f5b696d6d7fba119bb78bf9edf
I20260808 07:00:59.294992 20272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562: Bootstrap starting.
I20260808 07:00:59.295686 20272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562: Neither blocks nor log segments found. Creating new log.
I20260808 07:00:59.296011 20272 log.cc:824] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562: Log is configured to *not* fsync() on all Append() calls
I20260808 07:00:59.296705 20272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562: No bootstrap required, opened a new log
I20260808 07:00:59.298512 20272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683525f4377a4694afe95f78e49d9562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } }
I20260808 07:00:59.298666 20272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:00:59.298703 20272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683525f4377a4694afe95f78e49d9562, State: Initialized, Role: FOLLOWER
I20260808 07:00:59.298820 20272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683525f4377a4694afe95f78e49d9562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } }
I20260808 07:00:59.298879 20272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:00:59.298918 20272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:00:59.298964 20272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:00:59.299535 20272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683525f4377a4694afe95f78e49d9562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } }
I20260808 07:00:59.299641 20272 leader_election.cc:302] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 683525f4377a4694afe95f78e49d9562; no voters:
I20260808 07:00:59.299808 20272 leader_election.cc:288] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:00:59.299837 20277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:00:59.300026 20277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [term 1 LEADER]: Becoming Leader. State: Replica: 683525f4377a4694afe95f78e49d9562, State: Running, Role: LEADER
I20260808 07:00:59.300052 20272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:00:59.300115 20277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683525f4377a4694afe95f78e49d9562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } }
I20260808 07:00:59.300412 20277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 683525f4377a4694afe95f78e49d9562. Latest consensus state: current_term: 1 leader_uuid: "683525f4377a4694afe95f78e49d9562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683525f4377a4694afe95f78e49d9562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } } }
I20260808 07:00:59.300480 20277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:59.300680 20285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:00:59.300691 20278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "683525f4377a4694afe95f78e49d9562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683525f4377a4694afe95f78e49d9562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } } }
I20260808 07:00:59.300860 20278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562 [sys.catalog]: This master's current role is: LEADER
I20260808 07:00:59.301173 20285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 07:00:59 dist-test-slave-nj21 krb5kdc[20156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 07:00:59.302738 20285 catalog_manager.cc:1347] Generated new cluster ID: cbc468c754934771836288de15d6c9a7
I20260808 07:00:59.302789 20285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:00:59.305689 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:59.295160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38441 (local address 127.0.0.1:42063)
0808 07:00:59.295348 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:59.295358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:59.295395 (+ 37us) server_negotiation.cc:407] Connection header received
0808 07:00:59.295997 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:59.296006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:59.296064 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:59.296207 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:59.296860 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:59.298213 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:59.299322 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:59.299945 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:59.300360 (+ 415us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:59.302779 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:59.302783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:59.302783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:59.302794 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:59.304323 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:59.304668 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:59.304670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:59.304671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:59.304747 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:59.304955 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:59.304956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:59.304956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:59.305171 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:59.305309 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:59.305442 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 07:00:59.305534 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260808 07:00:59.306648 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:00:59.317179 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:00:59.311901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34239 (local address 127.0.0.1:42063)
0808 07:00:59.311932 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 07:00:59.311934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:00:59.312076 (+ 142us) server_negotiation.cc:407] Connection header received
0808 07:00:59.312127 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:00:59.312127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:00:59.312137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:00:59.312179 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:00:59.312631 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:59.313105 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:59.313747 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:00:59.313974 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:00:59.314025 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:00:59.315006 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:00:59.315007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:00:59.315008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:00:59.315014 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:00:59.316176 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:59.316496 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:59.316497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:59.316497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:59.316548 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:00:59.316726 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:00:59.316727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:00:59.316727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:00:59.316825 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:00:59.316910 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 07:00:59.317042 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 07:00:59.317099 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":15}
I20260808 07:00:59.330282 20285 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:00:59.330942 20285 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:00:59.338068 20285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 683525f4377a4694afe95f78e49d9562: Generated new TSK 0
I20260808 07:00:59.338229 20285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:00:59.341758 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 20175
W20260808 07:00:59.355690 20153 connection.cc:535] client connection to 127.0.0.1:42063 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42063): Connection reset by peer (error 104)
2026-08-08T07:00:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:00:59 dist-test-slave-nj21 krb5kdc[20319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:00:59 dist-test-slave-nj21 krb5kdc[20319](info): set up 2 sockets
Aug 08 07:00:59 dist-test-slave-nj21 krb5kdc[20319](info): commencing operation
krb5kdc: starting...
W20260808 07:01:00.478713 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:00 dist-test-slave-nj21 krb5kdc[20319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:00.587325 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:00.666962 20336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:00.667147 20336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:00.667168 20336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:00.668267 20336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:00.668304 20336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:00.668318 20336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:00.668336 20336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:00.668352 20336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:00.669750 20336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:00.669917 20336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:00.670086 20336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:00.674537 20344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:00.674510 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:00.674520 20343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:00.674611 20336 server_base.cc:1034] running on GCE node
I20260808 07:01:00.674993 20336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:00.675233 20336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:00.676354 20336 hybrid_clock.cc:648] HybridClock initialized: now 1786172460676340 us; error 29 us; skew 500 ppm
Aug 08 07:01:00 dist-test-slave-nj21 krb5kdc[20319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:00.678613 20336 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:00.679018 20336 webserver.cc:460] Webserver started at http://127.0.0.1:42645/ using document root <none> and password file <none>
I20260808 07:01:00.679226 20336 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:00.679273 20336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:00.679394 20336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:00.680413 20336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "da260204a3f9444aaa59d4e8baa07ad4"
format_stamp: "Formatted at 2026-08-08 07:01:00 on dist-test-slave-nj21"
server_key: "5eba49c519057ad227c91919c8ef00b3"
server_key_iv: "4dbe6e64063a5346d1a915cefc57a6a4"
server_key_version: "encryptionkey@0"
I20260808 07:01:00.680730 20336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "da260204a3f9444aaa59d4e8baa07ad4"
format_stamp: "Formatted at 2026-08-08 07:01:00 on dist-test-slave-nj21"
server_key: "5eba49c519057ad227c91919c8ef00b3"
server_key_iv: "4dbe6e64063a5346d1a915cefc57a6a4"
server_key_version: "encryptionkey@0"
I20260808 07:01:00.681893 20336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:01:00.682507 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:00.682652 20336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:01:00.682734 20336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "da260204a3f9444aaa59d4e8baa07ad4"
format_stamp: "Formatted at 2026-08-08 07:01:00 on dist-test-slave-nj21"
server_key: "5eba49c519057ad227c91919c8ef00b3"
server_key_iv: "4dbe6e64063a5346d1a915cefc57a6a4"
server_key_version: "encryptionkey@0"
I20260808 07:01:00.682790 20336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:00.693186 20336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:00.694659 20336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:00.694782 20336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:00.698671 20336 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38571
I20260808 07:01:00.698709 20449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38571 every 8 connection(s)
I20260808 07:01:00.698997 20336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:00.699767 20450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:00.701206 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 20336
I20260808 07:01:00.701321 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:00.701527 21617 external_mini_cluster.cc:1428] Setting key 749063ef332f50f80de33333e2c52a99
I20260808 07:01:00.702214 20450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4: Bootstrap starting.
I20260808 07:01:00.702771 20450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:00.703079 20450 log.cc:824] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:00.703763 20450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4: No bootstrap required, opened a new log
I20260808 07:01:00.705869 20450 raft_consensus.cc:348] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da260204a3f9444aaa59d4e8baa07ad4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260808 07:01:00.706019 20450 raft_consensus.cc:374] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:00.706063 20450 raft_consensus.cc:729] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da260204a3f9444aaa59d4e8baa07ad4, State: Initialized, Role: FOLLOWER
I20260808 07:01:00.706183 20450 consensus_queue.cc:260] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da260204a3f9444aaa59d4e8baa07ad4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260808 07:01:00.706256 20450 raft_consensus.cc:388] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:00.706276 20450 raft_consensus.cc:482] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:00.706321 20450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:00.706777 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:00.701988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46667 (local address 127.0.0.1:38571)
0808 07:01:00.702149 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:00.702157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:00.702180 (+ 23us) server_negotiation.cc:407] Connection header received
0808 07:01:00.702587 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:00.702594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:00.702636 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:00.702743 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:00.702975 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:00.704070 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:00.705147 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:00.705511 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:00.705580 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:00.706160 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:00.706163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:00.706164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:00.706337 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:00.706375 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:00.706388 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:00.706458 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:00.706567 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 07:01:00.706593 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260808 07:01:00.706885 20450 raft_consensus.cc:504] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da260204a3f9444aaa59d4e8baa07ad4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260808 07:01:00.707002 20450 leader_election.cc:302] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da260204a3f9444aaa59d4e8baa07ad4; no voters:
I20260808 07:01:00.707144 20450 leader_election.cc:288] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:00.707181 20459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:00.707326 20459 raft_consensus.cc:686] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [term 1 LEADER]: Becoming Leader. State: Replica: da260204a3f9444aaa59d4e8baa07ad4, State: Running, Role: LEADER
I20260808 07:01:00.707365 20450 sys_catalog.cc:564] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:00.707415 20459 consensus_queue.cc:237] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da260204a3f9444aaa59d4e8baa07ad4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260808 07:01:00.707731 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:00.707760 20461 sys_catalog.cc:455] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da260204a3f9444aaa59d4e8baa07ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da260204a3f9444aaa59d4e8baa07ad4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } } }
I20260808 07:01:00.707829 20461 sys_catalog.cc:458] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:00.707958 20461 sys_catalog.cc:455] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da260204a3f9444aaa59d4e8baa07ad4. Latest consensus state: current_term: 1 leader_uuid: "da260204a3f9444aaa59d4e8baa07ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da260204a3f9444aaa59d4e8baa07ad4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } } }
I20260808 07:01:00.708001 20461 sys_catalog.cc:458] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:00.708099 20468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:00.708572 20468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:00.710009 20468 catalog_manager.cc:1347] Generated new cluster ID: ce609710d61b4aeda500cfac33752641
I20260808 07:01:00.710059 20468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:00.711980 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:00.709334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49921 (local address 127.0.0.1:38571)
0808 07:01:00.709366 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:00.709368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:00.709376 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:01:00.709393 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:00.709393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:00.709401 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:00.709477 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:00.709753 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:00.710343 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:00.710982 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:00.711182 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:00.711233 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:00.711775 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:00.711776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:00.711776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:00.711780 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:00.711792 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:00.711795 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:00.711843 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:00.711893 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 07:01:00.711908 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 07:01:00.716934 20468 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:00.717375 20468 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:00.727787 20468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da260204a3f9444aaa59d4e8baa07ad4: Generated new TSK 0
I20260808 07:01:00.727927 20468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:00.731976 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 20336
W20260808 07:01:00.741278 20312 connection.cc:535] client connection to 127.0.0.1:38571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38571): Connection reset by peer (error 104)
2026-08-08T07:01:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:00 dist-test-slave-nj21 krb5kdc[20499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:00 dist-test-slave-nj21 krb5kdc[20499](info): set up 2 sockets
Aug 08 07:01:00 dist-test-slave-nj21 krb5kdc[20499](info): commencing operation
krb5kdc: starting...
W20260808 07:01:01.859577 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:01 dist-test-slave-nj21 krb5kdc[20499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:01.967721 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:02.045400 20519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:02.045599 20519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:02.045621 20519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:02.046689 20519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:02.046720 20519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:02.046732 20519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:02.046743 20519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:02.046756 20519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:02.048149 20519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:02.048326 20519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:02.048493 20519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:02.052938 20528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:02.052960 20525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:02.052954 20526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:02.052989 20519 server_base.cc:1034] running on GCE node
I20260808 07:01:02.053388 20519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:02.053678 20519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:02.054849 20519 hybrid_clock.cc:648] HybridClock initialized: now 1786172462054831 us; error 31 us; skew 500 ppm
Aug 08 07:01:02 dist-test-slave-nj21 krb5kdc[20499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:02.057301 20519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:02.057708 20519 webserver.cc:460] Webserver started at http://127.0.0.1:41971/ using document root <none> and password file <none>
I20260808 07:01:02.057898 20519 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:02.057945 20519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:02.058066 20519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:02.059077 20519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "f254c45a30074860b997c7599573ae6f"
format_stamp: "Formatted at 2026-08-08 07:01:02 on dist-test-slave-nj21"
server_key: "c4fb242ea1ad366c3ffd93af5db4154f"
server_key_iv: "1aa16e82e7a396cbdc69be69a4e137a5"
server_key_version: "encryptionkey@0"
I20260808 07:01:02.059397 20519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "f254c45a30074860b997c7599573ae6f"
format_stamp: "Formatted at 2026-08-08 07:01:02 on dist-test-slave-nj21"
server_key: "c4fb242ea1ad366c3ffd93af5db4154f"
server_key_iv: "1aa16e82e7a396cbdc69be69a4e137a5"
server_key_version: "encryptionkey@0"
I20260808 07:01:02.060685 20519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:02.061378 20536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:02.061533 20519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:02.061616 20519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "f254c45a30074860b997c7599573ae6f"
format_stamp: "Formatted at 2026-08-08 07:01:02 on dist-test-slave-nj21"
server_key: "c4fb242ea1ad366c3ffd93af5db4154f"
server_key_iv: "1aa16e82e7a396cbdc69be69a4e137a5"
server_key_version: "encryptionkey@0"
I20260808 07:01:02.061681 20519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:02.064829 20519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:02.066428 20519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:02.066551 20519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:02.070236 20519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39993
I20260808 07:01:02.070271 20624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39993 every 8 connection(s)
I20260808 07:01:02.070544 20519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:02.071311 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 20519
I20260808 07:01:02.071442 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:02.071537 20626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:02.071632 21617 external_mini_cluster.cc:1428] Setting key eed10e048b871c4615d7b985779e3f65
I20260808 07:01:02.073710 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:02.072100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38547 (local address 127.0.0.1:39993)
0808 07:01:02.072254 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:02.072261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:02.072285 (+ 24us) server_negotiation.cc:407] Connection header received
0808 07:01:02.072637 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:02.072643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:02.072665 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:02.072767 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:02.073333 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:02.073337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:02.073337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:02.073340 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:02.073373 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:02.073383 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:02.073436 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:02.073511 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 07:01:02.073541 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260808 07:01:02.074571 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:02.074669 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f: Bootstrap starting.
I20260808 07:01:02.075367 20626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:02.075704 20626 log.cc:824] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:02.076402 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f: No bootstrap required, opened a new log
I20260808 07:01:02.077023 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:02.076164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36939 (local address 127.0.0.1:39993)
0808 07:01:02.076200 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:02.076202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:02.076315 (+ 113us) server_negotiation.cc:407] Connection header received
0808 07:01:02.076342 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:02.076343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:02.076347 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:02.076417 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:02.076850 (+ 433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:02.076851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:02.076851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:02.076855 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:02.076867 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:02.076871 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:02.076928 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:02.076959 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 07:01:02.076970 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 07:01:02.078327 20626 raft_consensus.cc:348] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254c45a30074860b997c7599573ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260808 07:01:02.078455 20626 raft_consensus.cc:374] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:02.078488 20626 raft_consensus.cc:729] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f254c45a30074860b997c7599573ae6f, State: Initialized, Role: FOLLOWER
I20260808 07:01:02.078593 20626 consensus_queue.cc:260] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254c45a30074860b997c7599573ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260808 07:01:02.078681 20626 raft_consensus.cc:388] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:02.078724 20626 raft_consensus.cc:482] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:02.078771 20626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:02.079259 20626 raft_consensus.cc:504] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254c45a30074860b997c7599573ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260808 07:01:02.079352 20626 leader_election.cc:302] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f254c45a30074860b997c7599573ae6f; no voters:
I20260808 07:01:02.079499 20626 leader_election.cc:288] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:02.079531 20646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:02.079665 20646 raft_consensus.cc:686] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [term 1 LEADER]: Becoming Leader. State: Replica: f254c45a30074860b997c7599573ae6f, State: Running, Role: LEADER
I20260808 07:01:02.079761 20626 sys_catalog.cc:564] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:02.079774 20646 consensus_queue.cc:237] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254c45a30074860b997c7599573ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260808 07:01:02.080139 20648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f254c45a30074860b997c7599573ae6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254c45a30074860b997c7599573ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260808 07:01:02.080163 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f254c45a30074860b997c7599573ae6f. Latest consensus state: current_term: 1 leader_uuid: "f254c45a30074860b997c7599573ae6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f254c45a30074860b997c7599573ae6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260808 07:01:02.080300 20648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:02.080310 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:02.080670 20660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:02.081135 20660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:02.082535 20660 catalog_manager.cc:1347] Generated new cluster ID: 8fbf9724b3744b79ab8b80a1f06f2ddd
I20260808 07:01:02.082583 20660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:02.089651 20660 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:02.089782 20660 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:02.097152 20660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f254c45a30074860b997c7599573ae6f: Generated new TSK 0
I20260808 07:01:02.097301 20660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:02.105717 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 20519
2026-08-08T07:01:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:02 dist-test-slave-nj21 krb5kdc[20680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:02 dist-test-slave-nj21 krb5kdc[20680](info): set up 2 sockets
Aug 08 07:01:02 dist-test-slave-nj21 krb5kdc[20680](info): commencing operation
krb5kdc: starting...
W20260808 07:01:03.236163 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:03 dist-test-slave-nj21 krb5kdc[20680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:03.343739 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:03.422008 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:03.422192 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:03.422212 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:03.423275 20709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:03.423308 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:03.423321 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:03.423331 20709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:03.423345 20709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:03.424775 20709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:03.424965 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:03.425128 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:03.429700 20717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:03.429663 20719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:03.429700 20716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:03.429757 20709 server_base.cc:1034] running on GCE node
I20260808 07:01:03.430109 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:03.430352 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:03.431485 20709 hybrid_clock.cc:648] HybridClock initialized: now 1786172463431473 us; error 28 us; skew 500 ppm
Aug 08 07:01:03 dist-test-slave-nj21 krb5kdc[20680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:03.433777 20709 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:03.434196 20709 webserver.cc:460] Webserver started at http://127.0.0.1:42059/ using document root <none> and password file <none>
I20260808 07:01:03.434417 20709 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:03.434465 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:03.434592 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:03.435591 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "05bc0a2c51b2459cbb9d9082ed15b674"
format_stamp: "Formatted at 2026-08-08 07:01:03 on dist-test-slave-nj21"
server_key: "9c6c858b81d44dee39be4288dd66716c"
server_key_iv: "ff4810cf32431edf1e84e2abe4df5874"
server_key_version: "encryptionkey@0"
I20260808 07:01:03.435962 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "05bc0a2c51b2459cbb9d9082ed15b674"
format_stamp: "Formatted at 2026-08-08 07:01:03 on dist-test-slave-nj21"
server_key: "9c6c858b81d44dee39be4288dd66716c"
server_key_iv: "ff4810cf32431edf1e84e2abe4df5874"
server_key_version: "encryptionkey@0"
I20260808 07:01:03.437253 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:03.437973 20728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:03.438128 20709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:03.438200 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "05bc0a2c51b2459cbb9d9082ed15b674"
format_stamp: "Formatted at 2026-08-08 07:01:03 on dist-test-slave-nj21"
server_key: "9c6c858b81d44dee39be4288dd66716c"
server_key_iv: "ff4810cf32431edf1e84e2abe4df5874"
server_key_version: "encryptionkey@0"
I20260808 07:01:03.438277 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:03.458719 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:03.460363 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:03.460492 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:03.464221 20709 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45547
I20260808 07:01:03.464255 20818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45547 every 8 connection(s)
I20260808 07:01:03.464568 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:03.465350 20819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:03.467510 20819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674: Bootstrap starting.
I20260808 07:01:03.467576 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 20709
I20260808 07:01:03.467712 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:03.467907 21617 external_mini_cluster.cc:1428] Setting key b646afa1abfe67c4139468a2f74c5b46
I20260808 07:01:03.468207 20819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:03.468526 20819 log.cc:824] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:03.469187 20819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674: No bootstrap required, opened a new log
I20260808 07:01:03.471268 20819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc0a2c51b2459cbb9d9082ed15b674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45547 } }
I20260808 07:01:03.471429 20819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:03.471464 20819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bc0a2c51b2459cbb9d9082ed15b674, State: Initialized, Role: FOLLOWER
I20260808 07:01:03.471580 20819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc0a2c51b2459cbb9d9082ed15b674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45547 } }
I20260808 07:01:03.471648 20819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:03.471706 20819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:03.471756 20819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:03.472326 20819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc0a2c51b2459cbb9d9082ed15b674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45547 } }
I20260808 07:01:03.472419 20819 leader_election.cc:302] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05bc0a2c51b2459cbb9d9082ed15b674; no voters:
I20260808 07:01:03.472608 20819 leader_election.cc:288] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:03.472646 20827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:03.472815 20827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [term 1 LEADER]: Becoming Leader. State: Replica: 05bc0a2c51b2459cbb9d9082ed15b674, State: Running, Role: LEADER
I20260808 07:01:03.472913 20827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc0a2c51b2459cbb9d9082ed15b674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45547 } }
I20260808 07:01:03.473142 20819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:03.473270 20832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05bc0a2c51b2459cbb9d9082ed15b674. Latest consensus state: current_term: 1 leader_uuid: "05bc0a2c51b2459cbb9d9082ed15b674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc0a2c51b2459cbb9d9082ed15b674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45547 } } }
I20260808 07:01:03.473343 20832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:03.473557 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05bc0a2c51b2459cbb9d9082ed15b674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc0a2c51b2459cbb9d9082ed15b674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45547 } } }
I20260808 07:01:03.473660 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:03.473923 20837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:03.474390 20837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 07:01:03 dist-test-slave-nj21 krb5kdc[20680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 07:01:03.476356 20837 catalog_manager.cc:1347] Generated new cluster ID: bc3fe58fcce34d4ea0f2edf46419c3ea
I20260808 07:01:03.476409 20837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:03.478183 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:03.468346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56083 (local address 127.0.0.1:45547)
0808 07:01:03.468548 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:03.468557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:03.468589 (+ 32us) server_negotiation.cc:407] Connection header received
0808 07:01:03.469122 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:03.469130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:03.469178 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:03.469326 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:03.470047 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:03.470897 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:03.472016 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:03.472419 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:03.472524 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:03.475270 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:03.475275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:03.475275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:03.475287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:03.476862 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:03.477167 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:03.477170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:03.477170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:03.477248 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:03.477419 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:03.477419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:03.477420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:03.477632 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:03.477784 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:03.477872 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 07:01:03.478012 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260808 07:01:03.478991 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:03.484491 20837 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:03.485167 20837 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:03.489389 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:03.483933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47777 (local address 127.0.0.1:45547)
0808 07:01:03.483966 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:03.483968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:03.484110 (+ 142us) server_negotiation.cc:407] Connection header received
0808 07:01:03.484165 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:03.484166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:03.484175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:03.484246 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:03.484840 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:03.485340 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:03.485997 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:03.486205 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:03.486256 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:03.487239 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:03.487240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:03.487240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:03.487247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:03.488466 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:03.488748 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:03.488749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:03.488750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:03.488798 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:03.488961 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:03.488962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:03.488962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:03.489058 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:03.489134 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:03.489232 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 07:01:03.489308 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 07:01:03.498590 20837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05bc0a2c51b2459cbb9d9082ed15b674: Generated new TSK 0
I20260808 07:01:03.498781 20837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:03.507321 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 20709
2026-08-08T07:01:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:03 dist-test-slave-nj21 krb5kdc[20879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:03 dist-test-slave-nj21 krb5kdc[20879](info): set up 2 sockets
Aug 08 07:01:03 dist-test-slave-nj21 krb5kdc[20879](info): commencing operation
krb5kdc: starting...
W20260808 07:01:04.627544 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:04 dist-test-slave-nj21 krb5kdc[20879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:04.736115 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:04.813685 20903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:04.813868 20903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:04.813889 20903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:04.814987 20903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:04.815026 20903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:04.815039 20903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:04.815062 20903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:04.815088 20903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:04.816533 20903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:04.816740 20903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:04.816938 20903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:04.821372 20913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:04.821424 20909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:04.821578 20903 server_base.cc:1034] running on GCE node
W20260808 07:01:04.821376 20910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:04.821866 20903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:04.822141 20903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:04.823290 20903 hybrid_clock.cc:648] HybridClock initialized: now 1786172464823273 us; error 28 us; skew 500 ppm
Aug 08 07:01:04 dist-test-slave-nj21 krb5kdc[20879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:04.825588 20903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:04.825990 20903 webserver.cc:460] Webserver started at http://127.0.0.1:44305/ using document root <none> and password file <none>
I20260808 07:01:04.826175 20903 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:04.826220 20903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:04.826341 20903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:04.827318 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "8957409422d2496c9d24e92038eb1e49"
format_stamp: "Formatted at 2026-08-08 07:01:04 on dist-test-slave-nj21"
server_key: "7e86538f2177dc7600fcf09e2b554b55"
server_key_iv: "b75bc9d1e7fb5ba7912ea623583be310"
server_key_version: "encryptionkey@0"
I20260808 07:01:04.827646 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "8957409422d2496c9d24e92038eb1e49"
format_stamp: "Formatted at 2026-08-08 07:01:04 on dist-test-slave-nj21"
server_key: "7e86538f2177dc7600fcf09e2b554b55"
server_key_iv: "b75bc9d1e7fb5ba7912ea623583be310"
server_key_version: "encryptionkey@0"
I20260808 07:01:04.828833 20903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:04.829435 20921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:04.829593 20903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:04.829672 20903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "8957409422d2496c9d24e92038eb1e49"
format_stamp: "Formatted at 2026-08-08 07:01:04 on dist-test-slave-nj21"
server_key: "7e86538f2177dc7600fcf09e2b554b55"
server_key_iv: "b75bc9d1e7fb5ba7912ea623583be310"
server_key_version: "encryptionkey@0"
I20260808 07:01:04.829722 20903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:04.852039 20903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:04.853593 20903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:04.853693 20903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:04.857335 20903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41717
I20260808 07:01:04.857378 21003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41717 every 8 connection(s)
I20260808 07:01:04.857667 20903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:04.858460 21006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:04.859974 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 20903
I20260808 07:01:04.860103 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:04.860278 21617 external_mini_cluster.cc:1428] Setting key 54ac79a50b5df65c2ad6dab4017f617f
I20260808 07:01:04.861251 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49: Bootstrap starting.
I20260808 07:01:04.861886 21006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:04.862152 21006 log.cc:824] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:04.862844 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49: No bootstrap required, opened a new log
I20260808 07:01:04.865052 21006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8957409422d2496c9d24e92038eb1e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41717 } }
I20260808 07:01:04.865181 21006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:04.865202 21006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8957409422d2496c9d24e92038eb1e49, State: Initialized, Role: FOLLOWER
I20260808 07:01:04.865327 21006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8957409422d2496c9d24e92038eb1e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41717 } }
I20260808 07:01:04.865409 21006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:04.865439 21006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:04.865476 21006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:04.865639 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:04.860751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43033 (local address 127.0.0.1:41717)
0808 07:01:04.860925 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:04.860933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:04.860957 (+ 24us) server_negotiation.cc:407] Connection header received
0808 07:01:04.861405 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:04.861413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:04.861458 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:04.861566 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:04.861826 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:04.862854 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:04.863906 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:04.864307 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:04.864385 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:04.865013 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:04.865016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:04.865016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:04.865163 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:04.865200 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:04.865214 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:04.865275 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:04.865379 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 07:01:04.865445 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260808 07:01:04.866070 21006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8957409422d2496c9d24e92038eb1e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41717 } }
I20260808 07:01:04.866176 21006 leader_election.cc:302] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8957409422d2496c9d24e92038eb1e49; no voters:
I20260808 07:01:04.866348 21006 leader_election.cc:288] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:04.866374 21018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:04.866533 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:04.866555 21018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [term 1 LEADER]: Becoming Leader. State: Replica: 8957409422d2496c9d24e92038eb1e49, State: Running, Role: LEADER
I20260808 07:01:04.866585 21006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:04.866634 21018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8957409422d2496c9d24e92038eb1e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41717 } }
I20260808 07:01:04.866930 21018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8957409422d2496c9d24e92038eb1e49. Latest consensus state: current_term: 1 leader_uuid: "8957409422d2496c9d24e92038eb1e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8957409422d2496c9d24e92038eb1e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41717 } } }
I20260808 07:01:04.866982 21018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:04.867117 21022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8957409422d2496c9d24e92038eb1e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8957409422d2496c9d24e92038eb1e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41717 } } }
I20260808 07:01:04.867158 21022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:04.868094 21026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:04.868867 21026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:04.870288 21026 catalog_manager.cc:1347] Generated new cluster ID: 2cd35f2dd8fd4b059eea470f306f10fd
I20260808 07:01:04.870334 21026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:04.871181 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:04.868187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33807 (local address 127.0.0.1:41717)
0808 07:01:04.868216 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:04.868218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:04.868225 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:01:04.868244 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:04.868244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:04.868252 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:04.868310 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:04.868433 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:04.869056 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:04.870012 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:04.870292 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:04.870365 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:04.870964 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:04.870965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:04.870965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:04.870969 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:04.870983 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:04.870986 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:04.871027 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:04.871095 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 07:01:04.871114 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 07:01:04.879012 21026 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:04.879477 21026 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:04.883529 21026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8957409422d2496c9d24e92038eb1e49: Generated new TSK 0
I20260808 07:01:04.883694 21026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:04.888825 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 20903
2026-08-08T07:01:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:04 dist-test-slave-nj21 krb5kdc[21069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:04 dist-test-slave-nj21 krb5kdc[21069](info): set up 2 sockets
Aug 08 07:01:04 dist-test-slave-nj21 krb5kdc[21069](info): commencing operation
krb5kdc: starting...
W20260808 07:01:06.024178 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:06 dist-test-slave-nj21 krb5kdc[21069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:06.131948 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:06.211252 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:06.211445 21088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:06.211477 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:06.212692 21088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:06.212736 21088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:06.212756 21088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:06.212771 21088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:06.212787 21088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:06.214236 21088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:06.214473 21088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:06.214669 21088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:06.219246 21101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:06.219337 21088 server_base.cc:1034] running on GCE node
W20260808 07:01:06.219235 21100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:06.219225 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:06.219746 21088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:06.220049 21088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:06.221215 21088 hybrid_clock.cc:648] HybridClock initialized: now 1786172466221198 us; error 31 us; skew 500 ppm
Aug 08 07:01:06 dist-test-slave-nj21 krb5kdc[21069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:06.223487 21088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:06.223906 21088 webserver.cc:460] Webserver started at http://127.0.0.1:33221/ using document root <none> and password file <none>
I20260808 07:01:06.224092 21088 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:06.224136 21088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:06.224256 21088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:06.225243 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "74863406561b405c81503e22700ccd5a"
format_stamp: "Formatted at 2026-08-08 07:01:06 on dist-test-slave-nj21"
server_key: "7ac2756ae5df1658d3eaf3c744af7a46"
server_key_iv: "302a0c8075bf6963578072e07888c2de"
server_key_version: "encryptionkey@0"
I20260808 07:01:06.225569 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "74863406561b405c81503e22700ccd5a"
format_stamp: "Formatted at 2026-08-08 07:01:06 on dist-test-slave-nj21"
server_key: "7ac2756ae5df1658d3eaf3c744af7a46"
server_key_iv: "302a0c8075bf6963578072e07888c2de"
server_key_version: "encryptionkey@0"
I20260808 07:01:06.226776 21088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:01:06.227378 21113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:06.227527 21088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:01:06.227653 21088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "74863406561b405c81503e22700ccd5a"
format_stamp: "Formatted at 2026-08-08 07:01:06 on dist-test-slave-nj21"
server_key: "7ac2756ae5df1658d3eaf3c744af7a46"
server_key_iv: "302a0c8075bf6963578072e07888c2de"
server_key_version: "encryptionkey@0"
I20260808 07:01:06.227783 21088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:06.230927 21088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:06.232601 21088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:06.232721 21088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:06.236335 21088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38585
I20260808 07:01:06.236367 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38585 every 8 connection(s)
I20260808 07:01:06.236649 21088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:06.237496 21199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:06.239617 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a: Bootstrap starting.
I20260808 07:01:06.240154 21199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:06.240375 21199 log.cc:824] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:06.240959 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a: No bootstrap required, opened a new log
I20260808 07:01:06.242422 21199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74863406561b405c81503e22700ccd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38585 } }
I20260808 07:01:06.242545 21199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:06.242573 21199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74863406561b405c81503e22700ccd5a, State: Initialized, Role: FOLLOWER
I20260808 07:01:06.242676 21199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74863406561b405c81503e22700ccd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38585 } }
I20260808 07:01:06.242746 21199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:06.242774 21199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:06.242803 21199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:06.243258 21199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74863406561b405c81503e22700ccd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38585 } }
I20260808 07:01:06.243340 21199 leader_election.cc:302] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74863406561b405c81503e22700ccd5a; no voters:
I20260808 07:01:06.243459 21199 leader_election.cc:288] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:06.243507 21203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:06.243691 21203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [term 1 LEADER]: Becoming Leader. State: Replica: 74863406561b405c81503e22700ccd5a, State: Running, Role: LEADER
I20260808 07:01:06.243657 21199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:06.243839 21203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74863406561b405c81503e22700ccd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38585 } }
I20260808 07:01:06.244203 21206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74863406561b405c81503e22700ccd5a. Latest consensus state: current_term: 1 leader_uuid: "74863406561b405c81503e22700ccd5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74863406561b405c81503e22700ccd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38585 } } }
I20260808 07:01:06.244215 21204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74863406561b405c81503e22700ccd5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74863406561b405c81503e22700ccd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38585 } } }
I20260808 07:01:06.244360 21206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:06.244428 21204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:06.244714 21212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:06.245221 21212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:06.246050 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 21088
I20260808 07:01:06.246176 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:06.246359 21617 external_mini_cluster.cc:1428] Setting key 50e85f40cff53c72f9c0d9ed6e85506c
I20260808 07:01:06.246913 21212 catalog_manager.cc:1347] Generated new cluster ID: ca0cba8eae6d4a4cab17a3f23ae67bff
I20260808 07:01:06.247008 21212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:06.248278 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:06.246786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53985 (local address 127.0.0.1:38585)
0808 07:01:06.246953 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:06.246961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:06.247024 (+ 63us) server_negotiation.cc:407] Connection header received
0808 07:01:06.247128 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:06.247135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:06.247179 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:06.247305 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:06.247883 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:06.247886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:06.247887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:06.247890 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:06.247931 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:06.247946 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:06.248009 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:06.248068 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 07:01:06.248106 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260808 07:01:06.249123 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:06.251765 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:06.250665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33423 (local address 127.0.0.1:38585)
0808 07:01:06.250700 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:06.250702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:06.250810 (+ 108us) server_negotiation.cc:407] Connection header received
0808 07:01:06.250902 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:06.250902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:06.250907 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:06.250934 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:06.251527 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:06.251528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:06.251529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:06.251533 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:06.251544 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:06.251548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:06.251577 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:06.251694 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 07:01:06.251707 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 07:01:06.260210 21212 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:06.260365 21212 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:06.265429 21212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74863406561b405c81503e22700ccd5a: Generated new TSK 0
I20260808 07:01:06.265599 21212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:06.271467 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 21088
2026-08-08T07:01:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:06 dist-test-slave-nj21 krb5kdc[21248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:06 dist-test-slave-nj21 krb5kdc[21248](info): set up 2 sockets
Aug 08 07:01:06 dist-test-slave-nj21 krb5kdc[21248](info): commencing operation
krb5kdc: starting...
W20260808 07:01:07.390681 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:07 dist-test-slave-nj21 krb5kdc[21248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:07.497541 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:07.575286 21264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:07.575471 21264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:07.575492 21264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:07.576562 21264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:07.576596 21264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:07.576607 21264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:07.576618 21264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:07.576632 21264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:07.577996 21264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:07.578173 21264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:07.578334 21264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:07.582679 21273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:07.582677 21272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:07.582708 21276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:07.582782 21264 server_base.cc:1034] running on GCE node
I20260808 07:01:07.583189 21264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:07.583439 21264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:07.584569 21264 hybrid_clock.cc:648] HybridClock initialized: now 1786172467584555 us; error 26 us; skew 500 ppm
Aug 08 07:01:07 dist-test-slave-nj21 krb5kdc[21248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:07.586745 21264 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:07.587118 21264 webserver.cc:460] Webserver started at http://127.0.0.1:38211/ using document root <none> and password file <none>
I20260808 07:01:07.587308 21264 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:07.587353 21264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:07.587478 21264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:07.588471 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "05d2cb34ced04fe6b2d4b346d8572044"
format_stamp: "Formatted at 2026-08-08 07:01:07 on dist-test-slave-nj21"
server_key: "0d3ca30d652bb5187614d3de56cb4b42"
server_key_iv: "2640b07e3b497220318415c732177ebe"
server_key_version: "encryptionkey@0"
I20260808 07:01:07.588789 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "05d2cb34ced04fe6b2d4b346d8572044"
format_stamp: "Formatted at 2026-08-08 07:01:07 on dist-test-slave-nj21"
server_key: "0d3ca30d652bb5187614d3de56cb4b42"
server_key_iv: "2640b07e3b497220318415c732177ebe"
server_key_version: "encryptionkey@0"
I20260808 07:01:07.589964 21264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:01:07.590565 21285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:07.590710 21264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:07.590791 21264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "05d2cb34ced04fe6b2d4b346d8572044"
format_stamp: "Formatted at 2026-08-08 07:01:07 on dist-test-slave-nj21"
server_key: "0d3ca30d652bb5187614d3de56cb4b42"
server_key_iv: "2640b07e3b497220318415c732177ebe"
server_key_version: "encryptionkey@0"
I20260808 07:01:07.590865 21264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:07.608668 21264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:07.610193 21264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:07.610318 21264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:07.613924 21264 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40235
I20260808 07:01:07.613963 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40235 every 8 connection(s)
I20260808 07:01:07.614249 21264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:07.615051 21376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:07.617175 21376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044: Bootstrap starting.
I20260808 07:01:07.617674 21376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:07.617897 21376 log.cc:824] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:07.618494 21376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044: No bootstrap required, opened a new log
I20260808 07:01:07.620020 21376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d2cb34ced04fe6b2d4b346d8572044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260808 07:01:07.620146 21376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:07.620173 21376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d2cb34ced04fe6b2d4b346d8572044, State: Initialized, Role: FOLLOWER
I20260808 07:01:07.620265 21376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d2cb34ced04fe6b2d4b346d8572044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260808 07:01:07.620311 21376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:07.620329 21376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:07.620359 21376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:07.620791 21376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d2cb34ced04fe6b2d4b346d8572044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260808 07:01:07.620872 21376 leader_election.cc:302] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05d2cb34ced04fe6b2d4b346d8572044; no voters:
I20260808 07:01:07.620997 21376 leader_election.cc:288] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:07.621039 21379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:07.621201 21376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:07.621439 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 21264
I20260808 07:01:07.621541 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:07.621732 21617 external_mini_cluster.cc:1428] Setting key 271689274f019f325c3ef9f47ce16168
I20260808 07:01:07.621219 21379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [term 1 LEADER]: Becoming Leader. State: Replica: 05d2cb34ced04fe6b2d4b346d8572044, State: Running, Role: LEADER
I20260808 07:01:07.621975 21379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d2cb34ced04fe6b2d4b346d8572044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260808 07:01:07.622591 21380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05d2cb34ced04fe6b2d4b346d8572044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d2cb34ced04fe6b2d4b346d8572044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } } }
I20260808 07:01:07.622689 21380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:07.623004 21381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05d2cb34ced04fe6b2d4b346d8572044. Latest consensus state: current_term: 1 leader_uuid: "05d2cb34ced04fe6b2d4b346d8572044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d2cb34ced04fe6b2d4b346d8572044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } } }
I20260808 07:01:07.623070 21381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044 [sys.catalog]: This master's current role is: LEADER
W20260808 07:01:07.623085 21403 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:01:07.623129 21403 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:01:07.623342 21406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:07.623505 21406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:07.624950 21406 catalog_manager.cc:1347] Generated new cluster ID: 25e0ddac12414aac8432323d5378b619
I20260808 07:01:07.625002 21406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 07:01:07 dist-test-slave-nj21 krb5kdc[21248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 07:01:07.632817 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:07.622355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:43725 (local address 127.0.0.1:40235)
0808 07:01:07.622716 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:07.622725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:07.622762 (+ 37us) server_negotiation.cc:407] Connection header received
0808 07:01:07.622925 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:07.622933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:07.622995 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:07.623168 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:07.623911 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:07.624964 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:07.626022 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:07.626578 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:07.626664 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:07.629357 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:07.629361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:07.629361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:07.629372 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:07.631195 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:07.631593 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:07.631596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:07.631596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:07.631754 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:07.631961 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:07.631962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:07.631962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:07.632258 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:07.632418 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:07.632506 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 07:01:07.632606 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":61,"threads_started":1}
I20260808 07:01:07.633719 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:07.634945 21406 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:07.635495 21406 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:07.639611 21406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05d2cb34ced04fe6b2d4b346d8572044: Generated new TSK 0
I20260808 07:01:07.639777 21406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:07.642179 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:07.635735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:44699 (local address 127.0.0.1:40235)
0808 07:01:07.635785 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:07.635787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:07.635795 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:01:07.635813 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:07.635814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:07.635834 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:07.635886 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:07.636441 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:07.637236 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:07.638183 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:07.638507 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:07.638571 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:07.639873 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:07.639874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:07.639874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:07.639881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:07.641204 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:07.641518 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:07.641519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:07.641519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:07.641573 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:07.641737 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:07.641738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:07.641738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:07.641850 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:07.641942 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:07.642030 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 07:01:07.642092 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 07:01:07.644138 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 21264
W20260808 07:01:07.660583 21246 connection.cc:535] client connection to 127.0.0.1:40235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40235): Connection reset by peer (error 104)
2026-08-08T07:01:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:07 dist-test-slave-nj21 krb5kdc[21423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:07 dist-test-slave-nj21 krb5kdc[21423](info): set up 2 sockets
Aug 08 07:01:07 dist-test-slave-nj21 krb5kdc[21423](info): commencing operation
krb5kdc: starting...
W20260808 07:01:08.783353 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:08 dist-test-slave-nj21 krb5kdc[21423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:08.889596 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:08.967301 21444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:08.967485 21444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:08.967506 21444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:08.968632 21444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:08.968677 21444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:08.968690 21444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:08.968701 21444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:08.968714 21444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:08.970129 21444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:08.970304 21444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:08.970465 21444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:08.974823 21452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:08.974807 21455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:08.974901 21453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:08.975051 21444 server_base.cc:1034] running on GCE node
I20260808 07:01:08.975282 21444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:08.975509 21444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:08.976634 21444 hybrid_clock.cc:648] HybridClock initialized: now 1786172468976612 us; error 38 us; skew 500 ppm
Aug 08 07:01:08 dist-test-slave-nj21 krb5kdc[21423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:08.979001 21444 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:08.979491 21444 webserver.cc:460] Webserver started at http://127.0.0.1:43165/ using document root <none> and password file <none>
I20260808 07:01:08.979772 21444 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:08.979818 21444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:08.979951 21444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:08.980898 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "e05e753285004c9f8dbfaba6b19d2671"
format_stamp: "Formatted at 2026-08-08 07:01:08 on dist-test-slave-nj21"
server_key: "7503c676518ba29198ef075453f8c23e"
server_key_iv: "660f58f500258e4498b6366adbc9cba1"
server_key_version: "encryptionkey@0"
I20260808 07:01:08.981226 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "e05e753285004c9f8dbfaba6b19d2671"
format_stamp: "Formatted at 2026-08-08 07:01:08 on dist-test-slave-nj21"
server_key: "7503c676518ba29198ef075453f8c23e"
server_key_iv: "660f58f500258e4498b6366adbc9cba1"
server_key_version: "encryptionkey@0"
I20260808 07:01:08.982401 21444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:01:08.983044 21466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:08.983196 21444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:01:08.983268 21444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "e05e753285004c9f8dbfaba6b19d2671"
format_stamp: "Formatted at 2026-08-08 07:01:08 on dist-test-slave-nj21"
server_key: "7503c676518ba29198ef075453f8c23e"
server_key_iv: "660f58f500258e4498b6366adbc9cba1"
server_key_version: "encryptionkey@0"
I20260808 07:01:08.983332 21444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:08.997221 21444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:08.998848 21444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:08.998982 21444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:09.002764 21444 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38089
I20260808 07:01:09.002794 21546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38089 every 8 connection(s)
I20260808 07:01:09.003114 21444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:09.003454 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 21444
I20260808 07:01:09.003563 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:09.003767 21617 external_mini_cluster.cc:1428] Setting key 5f29ec5c7ba188bbb2c52d7e79d2e814
I20260808 07:01:09.004026 21547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:09.006666 21547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671: Bootstrap starting.
I20260808 07:01:09.007213 21547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:09.007467 21547 log.cc:824] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:09.008090 21547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671: No bootstrap required, opened a new log
I20260808 07:01:09.008823 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:09.004225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:40273 (local address 127.0.0.1:38089)
0808 07:01:09.004403 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:09.004410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:09.004436 (+ 26us) server_negotiation.cc:407] Connection header received
0808 07:01:09.005068 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:09.005075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:09.005123 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:09.005248 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:09.005470 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:09.006329 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:09.007139 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:09.007550 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:09.007651 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:09.008280 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:09.008284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:09.008284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:09.008437 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:09.008465 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:09.008475 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:09.008542 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:09.008633 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 07:01:09.008659 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260808 07:01:09.009764 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:09.010113 21547 raft_consensus.cc:348] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05e753285004c9f8dbfaba6b19d2671" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } }
I20260808 07:01:09.010241 21547 raft_consensus.cc:374] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:09.010273 21547 raft_consensus.cc:729] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05e753285004c9f8dbfaba6b19d2671, State: Initialized, Role: FOLLOWER
I20260808 07:01:09.010385 21547 consensus_queue.cc:260] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05e753285004c9f8dbfaba6b19d2671" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } }
I20260808 07:01:09.010462 21547 raft_consensus.cc:388] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:09.010493 21547 raft_consensus.cc:482] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:09.010540 21547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:09.011070 21547 raft_consensus.cc:504] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05e753285004c9f8dbfaba6b19d2671" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } }
I20260808 07:01:09.011206 21547 leader_election.cc:302] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05e753285004c9f8dbfaba6b19d2671; no voters:
I20260808 07:01:09.011353 21547 leader_election.cc:288] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:09.011402 21567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:09.011582 21547 sys_catalog.cc:564] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:09.011605 21567 raft_consensus.cc:686] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [term 1 LEADER]: Becoming Leader. State: Replica: e05e753285004c9f8dbfaba6b19d2671, State: Running, Role: LEADER
I20260808 07:01:09.011804 21567 consensus_queue.cc:237] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05e753285004c9f8dbfaba6b19d2671" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } }
I20260808 07:01:09.012130 21568 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e05e753285004c9f8dbfaba6b19d2671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05e753285004c9f8dbfaba6b19d2671" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } } }
I20260808 07:01:09.012197 21568 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:09.012359 21569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e05e753285004c9f8dbfaba6b19d2671. Latest consensus state: current_term: 1 leader_uuid: "e05e753285004c9f8dbfaba6b19d2671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05e753285004c9f8dbfaba6b19d2671" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } } }
I20260808 07:01:09.012393 21577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:09.012445 21569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:09.012897 21577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:09.014462 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:09.011120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:52697 (local address 127.0.0.1:38089)
0808 07:01:09.011170 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:09.011172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:09.011210 (+ 38us) server_negotiation.cc:407] Connection header received
0808 07:01:09.011267 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:09.011268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:09.011276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:09.011318 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:09.011496 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:09.012157 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:09.013124 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:09.013364 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:09.013395 (+ 31us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:09.014124 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:09.014125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:09.014125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:09.014129 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:09.014142 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:09.014146 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:09.014176 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:09.014354 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 07:01:09.014362 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 07:01:09.014370 21577 catalog_manager.cc:1347] Generated new cluster ID: dbf2db036dc54293bbb120a1d849edbe
I20260808 07:01:09.014670 21577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:09.028476 21577 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:09.028962 21577 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:09.032296 21577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e05e753285004c9f8dbfaba6b19d2671: Generated new TSK 0
I20260808 07:01:09.032490 21577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:09.039286 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 21444
W20260808 07:01:09.048957 21418 connection.cc:535] client connection to 127.0.0.1:38089 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38089): Connection reset by peer (error 104)
2026-08-08T07:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:09 dist-test-slave-nj21 krb5kdc[21600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:09 dist-test-slave-nj21 krb5kdc[21600](info): set up 2 sockets
Aug 08 07:01:09 dist-test-slave-nj21 krb5kdc[21600](info): commencing operation
krb5kdc: starting...
W20260808 07:01:10.172688 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:10 dist-test-slave-nj21 krb5kdc[21600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:10.279582 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:10.359182 21616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:10.359367 21616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:10.359397 21616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:10.360514 21616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:10.360554 21616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:10.360574 21616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:10.360590 21616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:10.360605 21616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:10.362033 21616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:10.362262 21616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:10.362453 21616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:10.366839 21628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:10.366839 21635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:10.366901 21630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:10.367095 21616 server_base.cc:1034] running on GCE node
I20260808 07:01:10.367296 21616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:10.367523 21616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:10.368667 21616 hybrid_clock.cc:648] HybridClock initialized: now 1786172470368642 us; error 41 us; skew 500 ppm
Aug 08 07:01:10 dist-test-slave-nj21 krb5kdc[21600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:10.370879 21616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:10.371246 21616 webserver.cc:460] Webserver started at http://127.0.0.1:36769/ using document root <none> and password file <none>
I20260808 07:01:10.371429 21616 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:10.371467 21616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:10.371551 21616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:10.372496 21616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "289eda78891c4d99bafe1ebeafdbcaa0"
format_stamp: "Formatted at 2026-08-08 07:01:10 on dist-test-slave-nj21"
server_key: "d70105c6aa1cbfd047805ada73778457"
server_key_iv: "58ef58b4926238effd2bdc61c9d4e685"
server_key_version: "encryptionkey@0"
I20260808 07:01:10.372776 21616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "289eda78891c4d99bafe1ebeafdbcaa0"
format_stamp: "Formatted at 2026-08-08 07:01:10 on dist-test-slave-nj21"
server_key: "d70105c6aa1cbfd047805ada73778457"
server_key_iv: "58ef58b4926238effd2bdc61c9d4e685"
server_key_version: "encryptionkey@0"
I20260808 07:01:10.373957 21616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:01:10.374625 21646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:10.374768 21616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:10.374858 21616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "289eda78891c4d99bafe1ebeafdbcaa0"
format_stamp: "Formatted at 2026-08-08 07:01:10 on dist-test-slave-nj21"
server_key: "d70105c6aa1cbfd047805ada73778457"
server_key_iv: "58ef58b4926238effd2bdc61c9d4e685"
server_key_version: "encryptionkey@0"
I20260808 07:01:10.374923 21616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:10.378002 21616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:10.379946 21616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:10.380071 21616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:10.383648 21616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36545
I20260808 07:01:10.383706 21747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36545 every 8 connection(s)
I20260808 07:01:10.383991 21616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:10.384863 21749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:10.386880 21749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0: Bootstrap starting.
I20260808 07:01:10.387408 21749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:10.387642 21749 log.cc:824] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:10.388281 21749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0: No bootstrap required, opened a new log
I20260808 07:01:10.389762 21749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289eda78891c4d99bafe1ebeafdbcaa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260808 07:01:10.389883 21749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:10.389925 21749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289eda78891c4d99bafe1ebeafdbcaa0, State: Initialized, Role: FOLLOWER
I20260808 07:01:10.390056 21749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289eda78891c4d99bafe1ebeafdbcaa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260808 07:01:10.390115 21749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:10.390136 21749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:10.390178 21749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:10.390652 21749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289eda78891c4d99bafe1ebeafdbcaa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260808 07:01:10.390735 21749 leader_election.cc:302] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 289eda78891c4d99bafe1ebeafdbcaa0; no voters:
I20260808 07:01:10.390877 21749 leader_election.cc:288] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:10.390923 21754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:10.391060 21754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [term 1 LEADER]: Becoming Leader. State: Replica: 289eda78891c4d99bafe1ebeafdbcaa0, State: Running, Role: LEADER
I20260808 07:01:10.391111 21749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:10.391146 21754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289eda78891c4d99bafe1ebeafdbcaa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260808 07:01:10.391506 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 289eda78891c4d99bafe1ebeafdbcaa0. Latest consensus state: current_term: 1 leader_uuid: "289eda78891c4d99bafe1ebeafdbcaa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289eda78891c4d99bafe1ebeafdbcaa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260808 07:01:10.391620 21756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "289eda78891c4d99bafe1ebeafdbcaa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289eda78891c4d99bafe1ebeafdbcaa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260808 07:01:10.391633 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:10.391798 21756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:10.392210 21771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:10.392701 21771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:10.393254 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 21616
I20260808 07:01:10.393363 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:10.393553 21617 external_mini_cluster.cc:1428] Setting key fd2b2fec803695fa6daa70f0595dae7d
I20260808 07:01:10.394179 21771 catalog_manager.cc:1347] Generated new cluster ID: 8923f816dbf647e0aa1ad856c4135775
I20260808 07:01:10.394217 21771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:10.397257 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:10.393998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:36565 (local address 127.0.0.1:36545)
0808 07:01:10.395799 (+ 1801us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:10.395807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:10.396003 (+ 196us) server_negotiation.cc:407] Connection header received
0808 07:01:10.396119 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:10.396126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:10.396155 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:10.396297 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:10.396862 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:10.396865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:10.396866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:10.396870 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:10.396910 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:10.396924 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:10.396991 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:10.397047 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 07:01:10.397089 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1742,"thread_start_us":67,"threads_started":1}
I20260808 07:01:10.398087 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:10.400594 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:10.399485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:33549 (local address 127.0.0.1:36545)
0808 07:01:10.399515 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:10.399516 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:10.399812 (+ 296us) server_negotiation.cc:407] Connection header received
0808 07:01:10.399843 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:10.399844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:10.399848 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:10.399907 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:10.400401 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:10.400403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:10.400403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:10.400408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:10.400422 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:10.400426 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:10.400482 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:10.400525 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 07:01:10.400537 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 07:01:10.411711 21771 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:10.411864 21771 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:10.415942 21771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 289eda78891c4d99bafe1ebeafdbcaa0: Generated new TSK 0
I20260808 07:01:10.416101 21771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:10.421662 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 21616
2026-08-08T07:01:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:10 dist-test-slave-nj21 krb5kdc[21793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:10 dist-test-slave-nj21 krb5kdc[21793](info): set up 2 sockets
Aug 08 07:01:10 dist-test-slave-nj21 krb5kdc[21793](info): commencing operation
krb5kdc: starting...
W20260808 07:01:11.546468 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:11 dist-test-slave-nj21 krb5kdc[21793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:11.652359 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:11.732178 21817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:11.732370 21817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:11.732402 21817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:11.733493 21817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:11.733533 21817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:11.733553 21817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:11.733568 21817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:11.733583 21817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:11.735011 21817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:11.735239 21817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:11.735435 21817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:11.739861 21827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:11.739869 21825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:11.739943 21832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:11.739974 21817 server_base.cc:1034] running on GCE node
I20260808 07:01:11.740283 21817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:11.740589 21817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:11.741726 21817 hybrid_clock.cc:648] HybridClock initialized: now 1786172471741714 us; error 27 us; skew 500 ppm
Aug 08 07:01:11 dist-test-slave-nj21 krb5kdc[21793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:11.743995 21817 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:11.744414 21817 webserver.cc:460] Webserver started at http://127.0.0.1:44851/ using document root <none> and password file <none>
I20260808 07:01:11.744599 21817 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:11.744645 21817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:11.744766 21817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:11.745739 21817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "d3eff167f90049c886157d110d7e985b"
format_stamp: "Formatted at 2026-08-08 07:01:11 on dist-test-slave-nj21"
server_key: "668dd99be3ac5b33674125913903dbb7"
server_key_iv: "f2d40e7a191fc399f38501d62429def5"
server_key_version: "encryptionkey@0"
I20260808 07:01:11.746052 21817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "d3eff167f90049c886157d110d7e985b"
format_stamp: "Formatted at 2026-08-08 07:01:11 on dist-test-slave-nj21"
server_key: "668dd99be3ac5b33674125913903dbb7"
server_key_iv: "f2d40e7a191fc399f38501d62429def5"
server_key_version: "encryptionkey@0"
I20260808 07:01:11.747244 21817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 07:01:11.747908 21843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:11.748059 21817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:11.748142 21817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "d3eff167f90049c886157d110d7e985b"
format_stamp: "Formatted at 2026-08-08 07:01:11 on dist-test-slave-nj21"
server_key: "668dd99be3ac5b33674125913903dbb7"
server_key_iv: "f2d40e7a191fc399f38501d62429def5"
server_key_version: "encryptionkey@0"
I20260808 07:01:11.748202 21817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:11.761857 21817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:11.763345 21817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:11.763473 21817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:11.767218 21817 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41247
I20260808 07:01:11.767249 21924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41247 every 8 connection(s)
I20260808 07:01:11.767547 21817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:11.768373 21925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:11.770498 21925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b: Bootstrap starting.
I20260808 07:01:11.771003 21925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:11.771216 21925 log.cc:824] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:11.771863 21925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b: No bootstrap required, opened a new log
I20260808 07:01:11.773392 21925 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3eff167f90049c886157d110d7e985b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41247 } }
I20260808 07:01:11.773531 21925 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:11.773552 21925 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3eff167f90049c886157d110d7e985b, State: Initialized, Role: FOLLOWER
I20260808 07:01:11.773651 21925 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3eff167f90049c886157d110d7e985b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41247 } }
I20260808 07:01:11.773698 21925 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:11.773720 21925 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:11.773748 21925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:11.774191 21925 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3eff167f90049c886157d110d7e985b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41247 } }
I20260808 07:01:11.774268 21925 leader_election.cc:302] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3eff167f90049c886157d110d7e985b; no voters:
I20260808 07:01:11.774380 21925 leader_election.cc:288] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:11.774428 21929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:11.774592 21929 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [term 1 LEADER]: Becoming Leader. State: Replica: d3eff167f90049c886157d110d7e985b, State: Running, Role: LEADER
I20260808 07:01:11.774596 21925 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:11.774767 21929 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3eff167f90049c886157d110d7e985b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41247 } }
I20260808 07:01:11.775134 21930 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3eff167f90049c886157d110d7e985b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3eff167f90049c886157d110d7e985b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41247 } } }
I20260808 07:01:11.775213 21930 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:11.775247 21932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3eff167f90049c886157d110d7e985b. Latest consensus state: current_term: 1 leader_uuid: "d3eff167f90049c886157d110d7e985b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3eff167f90049c886157d110d7e985b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41247 } } }
I20260808 07:01:11.775295 21932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:11.775449 21942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:11.775990 21942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:11.776285 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 21817
I20260808 07:01:11.776393 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:11.776577 21617 external_mini_cluster.cc:1428] Setting key 4ca7f3b1c98671194d6b0fbb1329f19d
I20260808 07:01:11.778034 21942 catalog_manager.cc:1347] Generated new cluster ID: fe8f2ffcc7bc4ca887eae53b8550a8dc
I20260808 07:01:11.778087 21942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 07:01:11 dist-test-slave-nj21 krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 07:01:11.785655 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:11.776974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:43523 (local address 127.0.0.1:41247)
0808 07:01:11.777132 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:11.777141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:11.777315 (+ 174us) server_negotiation.cc:407] Connection header received
0808 07:01:11.777450 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:11.777457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:11.777514 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:11.777642 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:11.778425 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:11.779305 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:11.780108 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:11.780520 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:11.780612 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:11.782721 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:11.782724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:11.782724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:11.782734 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:11.784185 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:11.784550 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:11.784552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:11.784553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:11.784642 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:11.784839 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:11.784840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:11.784841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:11.785061 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:11.785216 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:11.785337 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 07:01:11.785441 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260808 07:01:11.785679 21942 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:11.786499 21942 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:11.786581 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:11.792304 21942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3eff167f90049c886157d110d7e985b: Generated new TSK 0
I20260808 07:01:11.792474 21942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:11.793284 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:11.787869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:49775 (local address 127.0.0.1:41247)
0808 07:01:11.787901 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:11.787910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:11.788064 (+ 154us) server_negotiation.cc:407] Connection header received
0808 07:01:11.788113 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:11.788114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:11.788124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:11.788168 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:11.788630 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:11.789127 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:11.789794 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:11.790010 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:11.790061 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:11.791071 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:11.791072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:11.791072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:11.791078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:11.792278 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:11.792620 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:11.792621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:11.792621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:11.792672 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:11.792856 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:11.792857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:11.792857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:11.792955 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:11.793049 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:11.793154 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 07:01:11.793207 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 07:01:11.795167 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 21817
W20260808 07:01:11.804689 21790 connection.cc:535] client connection to 127.0.0.1:41247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41247): Connection reset by peer (error 104)
2026-08-08T07:01:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:11 dist-test-slave-nj21 krb5kdc[21971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:11 dist-test-slave-nj21 krb5kdc[21971](info): set up 2 sockets
Aug 08 07:01:11 dist-test-slave-nj21 krb5kdc[21971](info): commencing operation
krb5kdc: starting...
W20260808 07:01:12.927904 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:13 dist-test-slave-nj21 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:13.035880 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:13.114800 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:13.114997 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:13.115020 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:13.116300 21995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:13.116357 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:13.116382 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:13.116405 21995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:13.116480 21995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:13.118140 21995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:13.118371 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:13.118582 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:13.123005 22003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:13.123003 22005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:13.123080 22008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:13.123288 21995 server_base.cc:1034] running on GCE node
I20260808 07:01:13.123509 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:13.123822 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:13.124962 21995 hybrid_clock.cc:648] HybridClock initialized: now 1786172473124949 us; error 24 us; skew 500 ppm
Aug 08 07:01:13 dist-test-slave-nj21 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:13.127349 21995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:13.127806 21995 webserver.cc:460] Webserver started at http://127.0.0.1:34435/ using document root <none> and password file <none>
I20260808 07:01:13.128027 21995 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:13.128075 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:13.128209 21995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:13.129252 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "89ad2094c16546aeb71d991800f44d42"
format_stamp: "Formatted at 2026-08-08 07:01:13 on dist-test-slave-nj21"
server_key: "a9275adfae913235725e2206d56ae62c"
server_key_iv: "3ea9cdbb212c1c4bffb213d5f7c79b73"
server_key_version: "encryptionkey@0"
I20260808 07:01:13.129599 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "89ad2094c16546aeb71d991800f44d42"
format_stamp: "Formatted at 2026-08-08 07:01:13 on dist-test-slave-nj21"
server_key: "a9275adfae913235725e2206d56ae62c"
server_key_iv: "3ea9cdbb212c1c4bffb213d5f7c79b73"
server_key_version: "encryptionkey@0"
I20260808 07:01:13.130841 21995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:13.131536 22015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:13.131712 21995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:13.131837 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "89ad2094c16546aeb71d991800f44d42"
format_stamp: "Formatted at 2026-08-08 07:01:13 on dist-test-slave-nj21"
server_key: "a9275adfae913235725e2206d56ae62c"
server_key_iv: "3ea9cdbb212c1c4bffb213d5f7c79b73"
server_key_version: "encryptionkey@0"
I20260808 07:01:13.131906 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:13.160553 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:13.162581 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:13.162714 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:13.166443 21995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41879
I20260808 07:01:13.166478 22099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41879 every 8 connection(s)
I20260808 07:01:13.166795 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:13.167572 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:13.169683 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 21995
I20260808 07:01:13.169781 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:13.169855 22100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42: Bootstrap starting.
I20260808 07:01:13.169957 21617 external_mini_cluster.cc:1428] Setting key 830d70f584bb181f5874082cff40cc06
I20260808 07:01:13.170531 22100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:13.170835 22100 log.cc:824] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:13.171554 22100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42: No bootstrap required, opened a new log
I20260808 07:01:13.173698 22100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ad2094c16546aeb71d991800f44d42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41879 } }
I20260808 07:01:13.173854 22100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:13.173883 22100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ad2094c16546aeb71d991800f44d42, State: Initialized, Role: FOLLOWER
I20260808 07:01:13.174010 22100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ad2094c16546aeb71d991800f44d42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41879 } }
I20260808 07:01:13.174079 22100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:13.174113 22100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:13.174155 22100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:13.174747 22100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ad2094c16546aeb71d991800f44d42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41879 } }
I20260808 07:01:13.174836 22100 leader_election.cc:302] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ad2094c16546aeb71d991800f44d42; no voters:
I20260808 07:01:13.175020 22100 leader_election.cc:288] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:13.175055 22107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:13.175160 22107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [term 1 LEADER]: Becoming Leader. State: Replica: 89ad2094c16546aeb71d991800f44d42, State: Running, Role: LEADER
I20260808 07:01:13.175283 22100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:13.175294 22107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ad2094c16546aeb71d991800f44d42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41879 } }
I20260808 07:01:13.175606 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:13.170427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:33611 (local address 127.0.0.1:41879)
0808 07:01:13.170618 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:13.170627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:13.170666 (+ 39us) server_negotiation.cc:407] Connection header received
0808 07:01:13.171188 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:13.171195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:13.171252 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:13.171409 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:13.171566 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:13.172973 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:13.173701 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:13.174270 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:13.174381 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:13.174923 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:13.174928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:13.174928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:13.175099 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:13.175137 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:13.175152 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:13.175283 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:13.175381 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 07:01:13.175409 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260808 07:01:13.175722 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ad2094c16546aeb71d991800f44d42. Latest consensus state: current_term: 1 leader_uuid: "89ad2094c16546aeb71d991800f44d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ad2094c16546aeb71d991800f44d42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41879 } } }
I20260808 07:01:13.175822 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:13.175900 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89ad2094c16546aeb71d991800f44d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ad2094c16546aeb71d991800f44d42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41879 } } }
I20260808 07:01:13.176028 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:13.176119 22115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:13.176437 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:13.176620 22115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:13.178187 22115 catalog_manager.cc:1347] Generated new cluster ID: 5e98e1292593489499b485e30b45ec6e
I20260808 07:01:13.178236 22115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:13.181205 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:13.177920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:49405 (local address 127.0.0.1:41879)
0808 07:01:13.177955 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:13.177957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:13.178093 (+ 136us) server_negotiation.cc:407] Connection header received
0808 07:01:13.178182 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:13.178183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:13.178191 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:13.178216 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:13.178405 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:13.179085 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:13.180118 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:13.180407 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:13.180462 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:13.181007 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:13.181008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:13.181008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:13.181011 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:13.181021 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:13.181024 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:13.181065 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:13.181131 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 07:01:13.181139 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 07:01:13.196970 22115 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:13.197438 22115 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:13.203363 22115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89ad2094c16546aeb71d991800f44d42: Generated new TSK 0
I20260808 07:01:13.203518 22115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:13.210188 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 21995
W20260808 07:01:13.221117 21969 connection.cc:535] client connection to 127.0.0.1:41879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41879): Connection reset by peer (error 104)
2026-08-08T07:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:13 dist-test-slave-nj21 krb5kdc[22145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:13 dist-test-slave-nj21 krb5kdc[22145](info): set up 2 sockets
Aug 08 07:01:13 dist-test-slave-nj21 krb5kdc[22145](info): commencing operation
krb5kdc: starting...
W20260808 07:01:14.337522 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:14 dist-test-slave-nj21 krb5kdc[22145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 07:01:14.444491 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:14.521754 22162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:14.521931 22162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:14.521951 22162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:14.522986 22162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:14.523016 22162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:14.523028 22162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:14.523039 22162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:14.523053 22162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:14.524463 22162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:14.524641 22162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:14.524798 22162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:14.529140 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:14.529136 22176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:14.529136 22178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:14.529367 22162 server_base.cc:1034] running on GCE node
I20260808 07:01:14.529590 22162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:14.529807 22162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:14.530934 22162 hybrid_clock.cc:648] HybridClock initialized: now 1786172474530932 us; error 34 us; skew 500 ppm
Aug 08 07:01:14 dist-test-slave-nj21 krb5kdc[22145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:14.533192 22162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 07:01:14.533581 22162 webserver.cc:460] Webserver started at http://127.0.0.1:45695/ using document root <none> and password file <none>
I20260808 07:01:14.533772 22162 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:14.533813 22162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:14.533941 22162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:14.534864 22162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "5454962b6a0c42afa75afe47514eb0ab"
format_stamp: "Formatted at 2026-08-08 07:01:14 on dist-test-slave-nj21"
server_key: "c19499b54cd10a382f0d27dbf49d5553"
server_key_iv: "a703d57fbd31642b389883f76fc28a55"
server_key_version: "encryptionkey@0"
I20260808 07:01:14.535216 22162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "5454962b6a0c42afa75afe47514eb0ab"
format_stamp: "Formatted at 2026-08-08 07:01:14 on dist-test-slave-nj21"
server_key: "c19499b54cd10a382f0d27dbf49d5553"
server_key_iv: "a703d57fbd31642b389883f76fc28a55"
server_key_version: "encryptionkey@0"
I20260808 07:01:14.536501 22162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 07:01:14.537132 22187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:14.537287 22162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:01:14.537369 22162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "5454962b6a0c42afa75afe47514eb0ab"
format_stamp: "Formatted at 2026-08-08 07:01:14 on dist-test-slave-nj21"
server_key: "c19499b54cd10a382f0d27dbf49d5553"
server_key_iv: "a703d57fbd31642b389883f76fc28a55"
server_key_version: "encryptionkey@0"
I20260808 07:01:14.537433 22162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:14.540731 22162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:14.542675 22162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:14.542799 22162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:14.546384 22162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35717
I20260808 07:01:14.546413 22265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35717 every 8 connection(s)
I20260808 07:01:14.546687 22162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:14.547564 22266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:14.547957 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22162
I20260808 07:01:14.548058 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:14.548229 21617 external_mini_cluster.cc:1428] Setting key ebbeb39f66fb201205270df1deb77f79
I20260808 07:01:14.550304 22266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab: Bootstrap starting.
I20260808 07:01:14.550542 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:14.548694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:39739 (local address 127.0.0.1:35717)
0808 07:01:14.548908 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:14.548916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:14.548953 (+ 37us) server_negotiation.cc:407] Connection header received
0808 07:01:14.549417 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:14.549425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:14.549469 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:14.549618 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:14.550112 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:14.550116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:14.550116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:14.550120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:14.550158 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:14.550170 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:14.550240 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:14.550332 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 07:01:14.550371 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":111,"threads_started":1}
I20260808 07:01:14.550949 22266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:14.551182 22266 log.cc:824] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:14.551339 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:14.551819 22266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab: No bootstrap required, opened a new log
I20260808 07:01:14.553602 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:14.552714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:36037 (local address 127.0.0.1:35717)
0808 07:01:14.552745 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:14.552747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:14.552855 (+ 108us) server_negotiation.cc:407] Connection header received
0808 07:01:14.552937 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:14.552938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:14.552941 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:14.552984 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:14.553445 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:14.553446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:14.553446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:14.553449 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:14.553458 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:14.553461 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:14.553496 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:14.553555 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 07:01:14.553564 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 07:01:14.553854 22266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5454962b6a0c42afa75afe47514eb0ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260808 07:01:14.553985 22266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:14.554018 22266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5454962b6a0c42afa75afe47514eb0ab, State: Initialized, Role: FOLLOWER
I20260808 07:01:14.554131 22266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5454962b6a0c42afa75afe47514eb0ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260808 07:01:14.554204 22266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:14.554227 22266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:14.554273 22266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:14.554755 22266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5454962b6a0c42afa75afe47514eb0ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260808 07:01:14.554843 22266 leader_election.cc:302] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5454962b6a0c42afa75afe47514eb0ab; no voters:
I20260808 07:01:14.554991 22266 leader_election.cc:288] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:14.555027 22281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:14.555159 22281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5454962b6a0c42afa75afe47514eb0ab, State: Running, Role: LEADER
I20260808 07:01:14.555208 22266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:14.555269 22281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5454962b6a0c42afa75afe47514eb0ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260808 07:01:14.555596 22283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5454962b6a0c42afa75afe47514eb0ab. Latest consensus state: current_term: 1 leader_uuid: "5454962b6a0c42afa75afe47514eb0ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5454962b6a0c42afa75afe47514eb0ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } } }
I20260808 07:01:14.555713 22283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:14.555589 22282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5454962b6a0c42afa75afe47514eb0ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5454962b6a0c42afa75afe47514eb0ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } } }
I20260808 07:01:14.555843 22282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab [sys.catalog]: This master's current role is: LEADER
W20260808 07:01:14.556821 22303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:01:14.556862 22303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:01:14.556923 22291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:14.557090 22291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:14.558351 22291 catalog_manager.cc:1347] Generated new cluster ID: ecb168e01b3940e8af6c17dfd2af3034
I20260808 07:01:14.558395 22291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:14.573115 22291 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:14.573243 22291 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:14.577073 22291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5454962b6a0c42afa75afe47514eb0ab: Generated new TSK 0
I20260808 07:01:14.577207 22291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:14.585181 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22162
2026-08-08T07:01:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:14 dist-test-slave-nj21 krb5kdc[22308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:14 dist-test-slave-nj21 krb5kdc[22308](info): set up 2 sockets
Aug 08 07:01:14 dist-test-slave-nj21 krb5kdc[22308](info): commencing operation
krb5kdc: starting...
W20260808 07:01:15.716090 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:15 dist-test-slave-nj21 krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:01:15.824595 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46231
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:15.902920 22326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:15.903121 22326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:15.903156 22326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:15.904299 22326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:15.904338 22326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:15.904359 22326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:15.904374 22326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:15.904390 22326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:15.905840 22326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:46231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:15.906068 22326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:15.906262 22326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:15.910748 22335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:15.910775 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:15.910804 22343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:15.910835 22326 server_base.cc:1034] running on GCE node
I20260808 07:01:15.911171 22326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:15.911429 22326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:15.912571 22326 hybrid_clock.cc:648] HybridClock initialized: now 1786172475912550 us; error 34 us; skew 500 ppm
Aug 08 07:01:15 dist-test-slave-nj21 krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172475, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:15.917744 22326 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:01:15.918160 22326 webserver.cc:460] Webserver started at http://127.21.28.126:41901/ using document root <none> and password file <none>
I20260808 07:01:15.918344 22326 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:15.918388 22326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:15.918527 22326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:15.919487 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "f6466202ab244921ba0463dd948d79c7"
format_stamp: "Formatted at 2026-08-08 07:01:15 on dist-test-slave-nj21"
server_key: "afc0babcae4bf4240d170df812e46d48"
server_key_iv: "7ccda189d89288f445db53dd29e88831"
server_key_version: "encryptionkey@0"
I20260808 07:01:15.919818 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "f6466202ab244921ba0463dd948d79c7"
format_stamp: "Formatted at 2026-08-08 07:01:15 on dist-test-slave-nj21"
server_key: "afc0babcae4bf4240d170df812e46d48"
server_key_iv: "7ccda189d89288f445db53dd29e88831"
server_key_version: "encryptionkey@0"
I20260808 07:01:15.920997 22326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:01:15.921603 22352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:15.921752 22326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:01:15.921833 22326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "f6466202ab244921ba0463dd948d79c7"
format_stamp: "Formatted at 2026-08-08 07:01:15 on dist-test-slave-nj21"
server_key: "afc0babcae4bf4240d170df812e46d48"
server_key_iv: "7ccda189d89288f445db53dd29e88831"
server_key_version: "encryptionkey@0"
I20260808 07:01:15.921890 22326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:15.939071 22326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:15.940711 22326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:15.940834 22326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:15.944473 22326 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:46231
I20260808 07:01:15.944505 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:46231 every 8 connection(s)
I20260808 07:01:15.944795 22326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:15.945593 22430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:15.947726 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7: Bootstrap starting.
I20260808 07:01:15.948263 22430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:15.948510 22430 log.cc:824] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:15.948522 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22326
I20260808 07:01:15.948612 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:15.948787 21617 external_mini_cluster.cc:1428] Setting key 85ea90968461de0e273d27d238ce4762
I20260808 07:01:15.949162 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7: No bootstrap required, opened a new log
I20260808 07:01:15.950875 22430 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6466202ab244921ba0463dd948d79c7" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46231 } }
I20260808 07:01:15.951011 22430 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:15.951045 22430 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6466202ab244921ba0463dd948d79c7, State: Initialized, Role: FOLLOWER
I20260808 07:01:15.951157 22430 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6466202ab244921ba0463dd948d79c7" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46231 } }
I20260808 07:01:15.951217 22430 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:15.951247 22430 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:15.951292 22430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:15.951815 22430 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6466202ab244921ba0463dd948d79c7" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46231 } }
I20260808 07:01:15.951920 22430 leader_election.cc:302] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6466202ab244921ba0463dd948d79c7; no voters:
I20260808 07:01:15.952070 22430 leader_election.cc:288] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:15.952106 22435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:15.952262 22435 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [term 1 LEADER]: Becoming Leader. State: Replica: f6466202ab244921ba0463dd948d79c7, State: Running, Role: LEADER
I20260808 07:01:15.952314 22430 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:15.952425 22435 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6466202ab244921ba0463dd948d79c7" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46231 } }
I20260808 07:01:15.952750 22436 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6466202ab244921ba0463dd948d79c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6466202ab244921ba0463dd948d79c7" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46231 } } }
I20260808 07:01:15.952827 22436 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:15.952963 22438 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6466202ab244921ba0463dd948d79c7. Latest consensus state: current_term: 1 leader_uuid: "f6466202ab244921ba0463dd948d79c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6466202ab244921ba0463dd948d79c7" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 46231 } } }
I20260808 07:01:15.953012 22438 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:15.953470 22445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:15.953928 22445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:15.955449 22445 catalog_manager.cc:1347] Generated new cluster ID: b8900970eabb40fd875f48191e45b210
I20260808 07:01:15.955508 22445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 07:01:15 dist-test-slave-nj21 krb5kdc[22308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:01:15.959264 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:15.949222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55137 (local address 127.21.28.126:46231)
0808 07:01:15.949379 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:15.949388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:15.949432 (+ 44us) server_negotiation.cc:407] Connection header received
0808 07:01:15.950266 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:15.950273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:15.950317 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:15.950447 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:15.951035 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:15.952371 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:15.953358 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:15.953913 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:15.954005 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:15.956416 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:15.956420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:15.956420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:15.956431 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:15.957840 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:15.958179 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:15.958182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:15.958183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:15.958258 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:15.958451 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:15.958452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:15.958452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:15.958659 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:15.958814 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:15.958941 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 07:01:15.959032 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260808 07:01:15.960124 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:15.966776 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:15.961549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38633 (local address 127.21.28.126:46231)
0808 07:01:15.961582 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:15.961583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:15.961735 (+ 152us) server_negotiation.cc:407] Connection header received
0808 07:01:15.961785 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:15.961786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:15.961794 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:15.961837 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:15.962278 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:15.962734 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:15.963373 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:15.963574 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:15.963625 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:15.964626 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:15.964627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:15.964627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:15.964633 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:15.965778 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:15.966093 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:15.966094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:15.966095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:15.966141 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:15.966322 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:15.966322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:15.966323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:15.966424 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:15.966505 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:15.966640 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 07:01:15.966695 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 07:01:15.967499 22445 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:15.968467 22445 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:15.973234 22445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6466202ab244921ba0463dd948d79c7: Generated new TSK 0
I20260808 07:01:15.973387 22445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:15.977659 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22326
W20260808 07:01:15.992832 22306 connection.cc:535] client connection to 127.21.28.126:46231 recv error: Network error: failed to read from TLS socket (remote: 127.21.28.126:46231): Connection reset by peer (error 104)
2026-08-08T07:01:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:16 dist-test-slave-nj21 krb5kdc[22471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:16 dist-test-slave-nj21 krb5kdc[22471](info): set up 2 sockets
Aug 08 07:01:16 dist-test-slave-nj21 krb5kdc[22471](info): commencing operation
krb5kdc: starting...
W20260808 07:01:17.105410 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:17 dist-test-slave-nj21 krb5kdc[22471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:01:17.216476 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:42723
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:42723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:17.294077 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:17.294255 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:17.294276 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:17.295400 22489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:17.295449 22489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:17.295468 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:17.295485 22489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:17.295500 22489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:17.297142 22489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:42723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:42723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:17.297354 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:17.297564 22489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:17.301968 22502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:17.301995 22500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:17.302039 22498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:17.302093 22489 server_base.cc:1034] running on GCE node
I20260808 07:01:17.302358 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:17.302604 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:17.303720 22489 hybrid_clock.cc:648] HybridClock initialized: now 1786172477303705 us; error 27 us; skew 500 ppm
Aug 08 07:01:17 dist-test-slave-nj21 krb5kdc[22471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172477, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:17.305959 22489 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:01:17.306363 22489 webserver.cc:460] Webserver started at http://127.21.28.126:46143/ using document root <none> and password file <none>
I20260808 07:01:17.306581 22489 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:17.306627 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:17.306751 22489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:17.307699 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "47e6bddc9321469f8fbec18e9dbdf851"
format_stamp: "Formatted at 2026-08-08 07:01:17 on dist-test-slave-nj21"
server_key: "31034da886b9520b941807cad2b3818b"
server_key_iv: "e480d5588c046d7d3494913a2071f364"
server_key_version: "encryptionkey@0"
I20260808 07:01:17.308034 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "47e6bddc9321469f8fbec18e9dbdf851"
format_stamp: "Formatted at 2026-08-08 07:01:17 on dist-test-slave-nj21"
server_key: "31034da886b9520b941807cad2b3818b"
server_key_iv: "e480d5588c046d7d3494913a2071f364"
server_key_version: "encryptionkey@0"
I20260808 07:01:17.309201 22489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 07:01:17.309823 22509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:17.309981 22489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 07:01:17.310063 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "47e6bddc9321469f8fbec18e9dbdf851"
format_stamp: "Formatted at 2026-08-08 07:01:17 on dist-test-slave-nj21"
server_key: "31034da886b9520b941807cad2b3818b"
server_key_iv: "e480d5588c046d7d3494913a2071f364"
server_key_version: "encryptionkey@0"
I20260808 07:01:17.310127 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:17.318617 22489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:17.320137 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:17.320255 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:17.323848 22489 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:42723
I20260808 07:01:17.323877 22587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:42723 every 8 connection(s)
I20260808 07:01:17.324163 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:17.324894 22588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:17.326881 22588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851: Bootstrap starting.
I20260808 07:01:17.327513 22588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:17.327838 22588 log.cc:824] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:17.328639 22588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851: No bootstrap required, opened a new log
I20260808 07:01:17.330291 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22489
I20260808 07:01:17.330408 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:17.330588 21617 external_mini_cluster.cc:1428] Setting key 1b296782ac937821be322de0f899aba1
I20260808 07:01:17.330629 22588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6bddc9321469f8fbec18e9dbdf851" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42723 } }
I20260808 07:01:17.330778 22588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:17.330814 22588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e6bddc9321469f8fbec18e9dbdf851, State: Initialized, Role: FOLLOWER
I20260808 07:01:17.330938 22588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6bddc9321469f8fbec18e9dbdf851" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42723 } }
I20260808 07:01:17.331009 22588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:17.331043 22588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:17.331080 22588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:17.331636 22588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6bddc9321469f8fbec18e9dbdf851" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42723 } }
I20260808 07:01:17.331755 22588 leader_election.cc:302] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47e6bddc9321469f8fbec18e9dbdf851; no voters:
I20260808 07:01:17.331897 22588 leader_election.cc:288] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:17.331928 22599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:17.332033 22599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [term 1 LEADER]: Becoming Leader. State: Replica: 47e6bddc9321469f8fbec18e9dbdf851, State: Running, Role: LEADER
I20260808 07:01:17.332129 22588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:17.332109 22599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6bddc9321469f8fbec18e9dbdf851" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42723 } }
I20260808 07:01:17.332481 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e6bddc9321469f8fbec18e9dbdf851. Latest consensus state: current_term: 1 leader_uuid: "47e6bddc9321469f8fbec18e9dbdf851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6bddc9321469f8fbec18e9dbdf851" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42723 } } }
I20260808 07:01:17.332566 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:17.332715 22600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47e6bddc9321469f8fbec18e9dbdf851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6bddc9321469f8fbec18e9dbdf851" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 42723 } } }
I20260808 07:01:17.332758 22600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:17.333523 22607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:17.333982 22607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:17.335623 22607 catalog_manager.cc:1347] Generated new cluster ID: b4e5a69772804837ad382d3dc055496a
I20260808 07:01:17.335696 22607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:17.336355 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:17.331163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42865 (local address 127.21.28.126:42723)
0808 07:01:17.331305 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:17.331312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:17.331337 (+ 25us) server_negotiation.cc:407] Connection header received
0808 07:01:17.331613 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:17.331618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:17.331657 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:17.331804 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:17.332040 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:17.333386 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:17.334417 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:17.334821 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:17.334937 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:17.335696 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:17.335699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:17.335700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:17.335887 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:17.335925 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:17.335939 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:17.336013 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:17.336143 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 07:01:17.336166 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260808 07:01:17.337335 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:17.342901 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:17.338781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57553 (local address 127.21.28.126:42723)
0808 07:01:17.338812 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:17.338814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:17.339822 (+ 1008us) server_negotiation.cc:407] Connection header received
0808 07:01:17.339867 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:17.339868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:17.339878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:17.339945 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:17.340041 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:17.340764 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:17.341708 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:17.342043 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:17.342112 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:17.342676 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:17.342677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:17.342678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:17.342682 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:17.342695 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:17.342700 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:17.342758 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:17.342812 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 07:01:17.342824 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 07:01:17.359262 22607 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:17.359829 22607 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:17.368394 22607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47e6bddc9321469f8fbec18e9dbdf851: Generated new TSK 0
I20260808 07:01:17.368556 22607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:17.377432 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22489
W20260808 07:01:17.389107 22467 connection.cc:535] client connection to 127.21.28.126:42723 recv error: Network error: failed to read from TLS socket (remote: 127.21.28.126:42723): Connection reset by peer (error 104)
2026-08-08T07:01:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:17 dist-test-slave-nj21 krb5kdc[22639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:17 dist-test-slave-nj21 krb5kdc[22639](info): set up 2 sockets
Aug 08 07:01:17 dist-test-slave-nj21 krb5kdc[22639](info): commencing operation
krb5kdc: starting...
W20260808 07:01:18.497846 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:18 dist-test-slave-nj21 krb5kdc[22639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:01:18.606392 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36263
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:18.691982 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:18.692253 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:18.692288 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:18.694056 22657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:18.694103 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:18.694123 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:18.694139 22657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:18.694154 22657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:18.695713 22657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:18.695963 22657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:18.696190 22657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:18.701351 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:18.701344 22669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:18.701426 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:18.701689 22657 server_base.cc:1034] running on GCE node
I20260808 07:01:18.701927 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:18.702194 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:18.703330 22657 hybrid_clock.cc:648] HybridClock initialized: now 1786172478703314 us; error 29 us; skew 500 ppm
Aug 08 07:01:18 dist-test-slave-nj21 krb5kdc[22639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172478, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:18.706079 22657 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:01:18.707059 22657 webserver.cc:460] Webserver started at http://127.21.28.126:32807/ using document root <none> and password file <none>
I20260808 07:01:18.707306 22657 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:18.707367 22657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:18.707497 22657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:18.708885 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "31d85331b1b7495882bfeab12257db06"
format_stamp: "Formatted at 2026-08-08 07:01:18 on dist-test-slave-nj21"
server_key: "90bd0e4c6bb82c5fba1bf3a0eb2fab27"
server_key_iv: "999018eeac31a33bbdcb44e2aa2b4afb"
server_key_version: "encryptionkey@0"
I20260808 07:01:18.709227 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "31d85331b1b7495882bfeab12257db06"
format_stamp: "Formatted at 2026-08-08 07:01:18 on dist-test-slave-nj21"
server_key: "90bd0e4c6bb82c5fba1bf3a0eb2fab27"
server_key_iv: "999018eeac31a33bbdcb44e2aa2b4afb"
server_key_version: "encryptionkey@0"
I20260808 07:01:18.710439 22657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:01:18.711052 22678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:18.711210 22657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:18.711301 22657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "31d85331b1b7495882bfeab12257db06"
format_stamp: "Formatted at 2026-08-08 07:01:18 on dist-test-slave-nj21"
server_key: "90bd0e4c6bb82c5fba1bf3a0eb2fab27"
server_key_iv: "999018eeac31a33bbdcb44e2aa2b4afb"
server_key_version: "encryptionkey@0"
I20260808 07:01:18.711373 22657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:18.714625 22657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:18.716367 22657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:18.716492 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:18.720108 22657 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:36263
I20260808 07:01:18.720135 22756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:36263 every 8 connection(s)
I20260808 07:01:18.720418 22657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:18.721324 22757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:18.721443 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22657
I20260808 07:01:18.721563 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:18.721735 21617 external_mini_cluster.cc:1428] Setting key ba972466419206759031d98ac105810d
I20260808 07:01:18.723934 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:18.722186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49445 (local address 127.21.28.126:36263)
0808 07:01:18.722346 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:18.722355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:18.722388 (+ 33us) server_negotiation.cc:407] Connection header received
0808 07:01:18.722812 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:18.722821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:18.722853 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:18.723006 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:18.723493 (+ 487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:18.723496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:18.723496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:18.723500 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:18.723540 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:18.723552 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:18.723615 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:18.723667 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 07:01:18.723748 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260808 07:01:18.724567 22757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06: Bootstrap starting.
I20260808 07:01:18.724798 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:18.725181 22757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:18.725428 22757 log.cc:824] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:18.726089 22757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06: No bootstrap required, opened a new log
I20260808 07:01:18.727005 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:18.726119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40473 (local address 127.21.28.126:36263)
0808 07:01:18.726154 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:18.726156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:18.726259 (+ 103us) server_negotiation.cc:407] Connection header received
0808 07:01:18.726293 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:18.726294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:18.726298 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:18.726362 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:18.726817 (+ 455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:18.726818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:18.726818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:18.726822 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:18.726836 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:18.726847 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:18.726907 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:18.726943 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 07:01:18.726954 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 07:01:18.728209 22757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d85331b1b7495882bfeab12257db06" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36263 } }
I20260808 07:01:18.728332 22757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:18.728366 22757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d85331b1b7495882bfeab12257db06, State: Initialized, Role: FOLLOWER
I20260808 07:01:18.728457 22757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d85331b1b7495882bfeab12257db06" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36263 } }
I20260808 07:01:18.728538 22757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:18.728564 22757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:18.728595 22757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:18.729039 22757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d85331b1b7495882bfeab12257db06" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36263 } }
I20260808 07:01:18.729126 22757 leader_election.cc:302] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d85331b1b7495882bfeab12257db06; no voters:
I20260808 07:01:18.729247 22757 leader_election.cc:288] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:18.729298 22770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:18.729466 22757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:18.729483 22770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [term 1 LEADER]: Becoming Leader. State: Replica: 31d85331b1b7495882bfeab12257db06, State: Running, Role: LEADER
I20260808 07:01:18.729609 22770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d85331b1b7495882bfeab12257db06" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36263 } }
I20260808 07:01:18.729924 22771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d85331b1b7495882bfeab12257db06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d85331b1b7495882bfeab12257db06" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36263 } } }
I20260808 07:01:18.729940 22772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d85331b1b7495882bfeab12257db06. Latest consensus state: current_term: 1 leader_uuid: "31d85331b1b7495882bfeab12257db06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d85331b1b7495882bfeab12257db06" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36263 } } }
I20260808 07:01:18.730099 22771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:18.730110 22772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:18.730481 22781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:18.731052 22781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:18.732367 22781 catalog_manager.cc:1347] Generated new cluster ID: 4d79300ecfb5420e895f68be38b1d8fd
I20260808 07:01:18.732425 22781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:18.750344 22781 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:18.750499 22781 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:18.757934 22781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31d85331b1b7495882bfeab12257db06: Generated new TSK 0
I20260808 07:01:18.758083 22781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:18.764230 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22657
2026-08-08T07:01:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:18 dist-test-slave-nj21 krb5kdc[22801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:18 dist-test-slave-nj21 krb5kdc[22801](info): set up 2 sockets
Aug 08 07:01:18 dist-test-slave-nj21 krb5kdc[22801](info): commencing operation
krb5kdc: starting...
W20260808 07:01:19.909134 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:19 dist-test-slave-nj21 krb5kdc[22801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:01:20.018443 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:35451
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:35451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:20.105237 22822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:20.105443 22822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:20.105475 22822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:20.106598 22822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:20.106638 22822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:20.106658 22822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:20.106673 22822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:20.106688 22822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:20.108134 22822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:35451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:20.108368 22822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:20.108569 22822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:20.113098 22829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:20.113077 22830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:20.113188 22822 server_base.cc:1034] running on GCE node
W20260808 07:01:20.113338 22833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:20.113565 22822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:20.113814 22822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:20.114946 22822 hybrid_clock.cc:648] HybridClock initialized: now 1786172480114930 us; error 29 us; skew 500 ppm
Aug 08 07:01:20 dist-test-slave-nj21 krb5kdc[22801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172480, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:20.117270 22822 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:01:20.117648 22822 webserver.cc:460] Webserver started at http://127.21.28.126:42051/ using document root <none> and password file <none>
I20260808 07:01:20.117831 22822 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:20.117877 22822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:20.117992 22822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:20.118949 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "eb6ed6a080a74233983c438911a51b6d"
format_stamp: "Formatted at 2026-08-08 07:01:20 on dist-test-slave-nj21"
server_key: "928aa6d378377065de4869dc9a5e8082"
server_key_iv: "2c33b995a16616bbeb6df056a4c22610"
server_key_version: "encryptionkey@0"
I20260808 07:01:20.119266 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "eb6ed6a080a74233983c438911a51b6d"
format_stamp: "Formatted at 2026-08-08 07:01:20 on dist-test-slave-nj21"
server_key: "928aa6d378377065de4869dc9a5e8082"
server_key_iv: "2c33b995a16616bbeb6df056a4c22610"
server_key_version: "encryptionkey@0"
I20260808 07:01:20.120524 22822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:20.121176 22844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:20.121326 22822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:20.121415 22822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "eb6ed6a080a74233983c438911a51b6d"
format_stamp: "Formatted at 2026-08-08 07:01:20 on dist-test-slave-nj21"
server_key: "928aa6d378377065de4869dc9a5e8082"
server_key_iv: "2c33b995a16616bbeb6df056a4c22610"
server_key_version: "encryptionkey@0"
I20260808 07:01:20.121471 22822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:20.139142 22822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:20.141129 22822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:20.141258 22822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:20.144901 22822 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:35451
I20260808 07:01:20.144941 22911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:35451 every 8 connection(s)
I20260808 07:01:20.145282 22822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:20.146199 22912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:20.148572 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d: Bootstrap starting.
I20260808 07:01:20.149118 22912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:20.149360 22912 log.cc:824] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:20.149993 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d: No bootstrap required, opened a new log
I20260808 07:01:20.151495 22912 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ed6a080a74233983c438911a51b6d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35451 } }
I20260808 07:01:20.151624 22912 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:20.151655 22912 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6ed6a080a74233983c438911a51b6d, State: Initialized, Role: FOLLOWER
I20260808 07:01:20.151806 22912 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ed6a080a74233983c438911a51b6d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35451 } }
I20260808 07:01:20.151865 22912 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:20.151896 22912 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:20.151942 22912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:20.152704 22912 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ed6a080a74233983c438911a51b6d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35451 } }
I20260808 07:01:20.152791 22912 leader_election.cc:302] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6ed6a080a74233983c438911a51b6d; no voters:
I20260808 07:01:20.152933 22912 leader_election.cc:288] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:20.152969 22918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:20.153105 22918 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [term 1 LEADER]: Becoming Leader. State: Replica: eb6ed6a080a74233983c438911a51b6d, State: Running, Role: LEADER
I20260808 07:01:20.153151 22912 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:20.153188 22918 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ed6a080a74233983c438911a51b6d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35451 } }
I20260808 07:01:20.153504 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6ed6a080a74233983c438911a51b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ed6a080a74233983c438911a51b6d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35451 } } }
I20260808 07:01:20.153523 22920 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6ed6a080a74233983c438911a51b6d. Latest consensus state: current_term: 1 leader_uuid: "eb6ed6a080a74233983c438911a51b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ed6a080a74233983c438911a51b6d" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 35451 } } }
I20260808 07:01:20.153607 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:20.153611 22920 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:20.153846 22929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:20.154349 22929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:20.155292 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22822
I20260808 07:01:20.155407 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:20.155579 21617 external_mini_cluster.cc:1428] Setting key b8a08cf9521d5a4ff46243f6b074aaa8
I20260808 07:01:20.156080 22929 catalog_manager.cc:1347] Generated new cluster ID: 6dc0192cf6da4cc8bb617ef2a274c144
I20260808 07:01:20.156129 22929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 07:01:20 dist-test-slave-nj21 krb5kdc[22801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:01:20.168188 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:20.156193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54883 (local address 127.21.28.126:35451)
0808 07:01:20.159957 (+ 3764us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:20.159965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:20.159988 (+ 23us) server_negotiation.cc:407] Connection header received
0808 07:01:20.160069 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:20.160074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:20.160111 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:20.160215 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:20.160841 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:20.162089 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:20.162794 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:20.163211 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:20.163301 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:20.165433 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:20.165437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:20.165438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:20.165449 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:20.166861 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:20.167162 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:20.167165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:20.167165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:20.167243 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:20.167415 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:20.167415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:20.167416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:20.167620 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:20.167764 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:20.167866 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 07:01:20.167993 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3727,"thread_start_us":63,"threads_started":1}
I20260808 07:01:20.168962 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:20.173926 22929 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:20.174558 22929 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:20.177044 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:20.170205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56675 (local address 127.21.28.126:35451)
0808 07:01:20.170244 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:20.170246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:20.171829 (+ 1583us) server_negotiation.cc:407] Connection header received
0808 07:01:20.171881 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:20.171882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:20.171891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:20.171935 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:20.172381 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:20.172861 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:20.173501 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:20.173801 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:20.173875 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:20.174927 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:20.174928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:20.174928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:20.174934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:20.176106 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:20.176384 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:20.176386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:20.176386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:20.176442 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:20.176602 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:20.176602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:20.176602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:20.176697 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:20.176797 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:20.176883 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 07:01:20.176964 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":20}
I20260808 07:01:20.186623 22929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6ed6a080a74233983c438911a51b6d: Generated new TSK 0
I20260808 07:01:20.186772 22929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:20.190892 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22822
2026-08-08T07:01:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:20 dist-test-slave-nj21 krb5kdc[22957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:20 dist-test-slave-nj21 krb5kdc[22957](info): set up 2 sockets
Aug 08 07:01:20 dist-test-slave-nj21 krb5kdc[22957](info): commencing operation
krb5kdc: starting...
W20260808 07:01:21.318593 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:21 dist-test-slave-nj21 krb5kdc[22957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:01:21.427593 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:40831
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:40831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:21.507997 22973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:21.508179 22973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:21.508199 22973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:21.509240 22973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:21.509269 22973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:21.509281 22973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:21.509292 22973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:21.509305 22973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:21.510720 22973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:40831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:40831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:21.510912 22973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:21.511076 22973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:21.515491 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:21.515497 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:21.515542 22973 server_base.cc:1034] running on GCE node
W20260808 07:01:21.515508 22980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:21.515961 22973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:21.516213 22973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:21.517345 22973 hybrid_clock.cc:648] HybridClock initialized: now 1786172481517335 us; error 28 us; skew 500 ppm
Aug 08 07:01:21 dist-test-slave-nj21 krb5kdc[22957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172481, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:21.522212 22973 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:01:21.522605 22973 webserver.cc:460] Webserver started at http://127.21.28.126:34815/ using document root <none> and password file <none>
I20260808 07:01:21.522796 22973 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:21.522842 22973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:21.522970 22973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:21.523991 22973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "890c8de4a75a44e4b62bd12d0a125bd4"
format_stamp: "Formatted at 2026-08-08 07:01:21 on dist-test-slave-nj21"
server_key: "1e794379f48e01ab39e50daeb19eaeb6"
server_key_iv: "969f217a143c79b5702cb6ae26da7702"
server_key_version: "encryptionkey@0"
I20260808 07:01:21.524299 22973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "890c8de4a75a44e4b62bd12d0a125bd4"
format_stamp: "Formatted at 2026-08-08 07:01:21 on dist-test-slave-nj21"
server_key: "1e794379f48e01ab39e50daeb19eaeb6"
server_key_iv: "969f217a143c79b5702cb6ae26da7702"
server_key_version: "encryptionkey@0"
I20260808 07:01:21.525537 22973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:01:21.526151 22994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:21.526289 22973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:21.526369 22973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "890c8de4a75a44e4b62bd12d0a125bd4"
format_stamp: "Formatted at 2026-08-08 07:01:21 on dist-test-slave-nj21"
server_key: "1e794379f48e01ab39e50daeb19eaeb6"
server_key_iv: "969f217a143c79b5702cb6ae26da7702"
server_key_version: "encryptionkey@0"
I20260808 07:01:21.526427 22973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:21.537545 22973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:21.539047 22973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:21.539170 22973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:21.542866 22973 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:40831
I20260808 07:01:21.542894 23066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:40831 every 8 connection(s)
I20260808 07:01:21.543187 22973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:21.544003 23067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:21.546123 23067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4: Bootstrap starting.
I20260808 07:01:21.546659 23067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:21.546897 23067 log.cc:824] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:21.547497 23067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4: No bootstrap required, opened a new log
I20260808 07:01:21.549057 23067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890c8de4a75a44e4b62bd12d0a125bd4" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40831 } }
I20260808 07:01:21.549185 23067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:21.549206 23067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890c8de4a75a44e4b62bd12d0a125bd4, State: Initialized, Role: FOLLOWER
I20260808 07:01:21.549309 23067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890c8de4a75a44e4b62bd12d0a125bd4" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40831 } }
I20260808 07:01:21.549382 23067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:21.549412 23067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:21.549453 23067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:21.550189 23067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890c8de4a75a44e4b62bd12d0a125bd4" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40831 } }
I20260808 07:01:21.550271 23067 leader_election.cc:302] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890c8de4a75a44e4b62bd12d0a125bd4; no voters:
I20260808 07:01:21.550383 23067 leader_election.cc:288] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:21.550438 23070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:21.550580 23070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [term 1 LEADER]: Becoming Leader. State: Replica: 890c8de4a75a44e4b62bd12d0a125bd4, State: Running, Role: LEADER
I20260808 07:01:21.550601 23067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:21.550662 23070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890c8de4a75a44e4b62bd12d0a125bd4" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40831 } }
I20260808 07:01:21.551019 23072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890c8de4a75a44e4b62bd12d0a125bd4. Latest consensus state: current_term: 1 leader_uuid: "890c8de4a75a44e4b62bd12d0a125bd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890c8de4a75a44e4b62bd12d0a125bd4" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40831 } } }
I20260808 07:01:21.551100 23072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:21.551254 23071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890c8de4a75a44e4b62bd12d0a125bd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890c8de4a75a44e4b62bd12d0a125bd4" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 40831 } } }
I20260808 07:01:21.551316 23071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:21.551568 23081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:21.551913 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 22973
I20260808 07:01:21.552021 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:21.552205 21617 external_mini_cluster.cc:1428] Setting key 34536953dea42b8113cf27849bb4849c
I20260808 07:01:21.552222 23081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:21.554139 23081 catalog_manager.cc:1347] Generated new cluster ID: 8d01afc27e9e4859ab78779747b2a089
I20260808 07:01:21.554184 23081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:21.558535 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:21.553314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43565 (local address 127.21.28.126:40831)
0808 07:01:21.553465 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:21.553473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:21.553502 (+ 29us) server_negotiation.cc:407] Connection header received
0808 07:01:21.553616 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:21.553623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:21.553678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:21.553811 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:21.554029 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:21.555373 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:21.556484 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:21.557063 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:21.557152 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:21.557772 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:21.557776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:21.557776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:21.557967 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:21.558001 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:21.558014 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:21.558070 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:21.558151 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 07:01:21.558206 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260808 07:01:21.559358 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:21.563949 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:21.560781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38411 (local address 127.21.28.126:40831)
0808 07:01:21.560822 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:21.560824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:21.560919 (+ 95us) server_negotiation.cc:407] Connection header received
0808 07:01:21.560961 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:21.560962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:21.560972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:21.561034 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:21.561142 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:21.561848 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:21.562765 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:21.563095 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:21.563159 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:21.563718 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:21.563719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:21.563719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:21.563724 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:21.563738 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:21.563743 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:21.563800 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:21.563848 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 07:01:21.563871 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 07:01:21.565727 23081 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:21.566154 23081 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:21.570076 23081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890c8de4a75a44e4b62bd12d0a125bd4: Generated new TSK 0
I20260808 07:01:21.570218 23081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:21.576041 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 22973
2026-08-08T07:01:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:21 dist-test-slave-nj21 krb5kdc[23109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:21 dist-test-slave-nj21 krb5kdc[23109](info): set up 2 sockets
Aug 08 07:01:21 dist-test-slave-nj21 krb5kdc[23109](info): commencing operation
krb5kdc: starting...
W20260808 07:01:22.708019 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:22 dist-test-slave-nj21 krb5kdc[23109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:01:22.816129 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:33851
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:33851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:22.893554 23129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:22.893730 23129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:22.893751 23129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:22.894798 23129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:22.894830 23129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:22.894843 23129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:22.894860 23129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:22.894876 23129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:22.896270 23129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:33851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:22.896441 23129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:22.896601 23129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:22.901058 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:22.901051 23141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:22.901073 23129 server_base.cc:1034] running on GCE node
W20260808 07:01:22.901068 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:22.901505 23129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:22.901784 23129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:22.902925 23129 hybrid_clock.cc:648] HybridClock initialized: now 1786172482902910 us; error 27 us; skew 500 ppm
Aug 08 07:01:22 dist-test-slave-nj21 krb5kdc[23109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172482, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:22.905208 23129 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:01:22.905598 23129 webserver.cc:460] Webserver started at http://127.21.28.126:43751/ using document root <none> and password file <none>
I20260808 07:01:22.905799 23129 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:22.905845 23129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:22.905972 23129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:22.906927 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "01fdb6de77974963a35477a7fe852281"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-nj21"
server_key: "81914d34a8101da941cc3112f29a1eef"
server_key_iv: "a2899b5ca1e348e4cf95b7d1c73088fb"
server_key_version: "encryptionkey@0"
I20260808 07:01:22.907232 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "01fdb6de77974963a35477a7fe852281"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-nj21"
server_key: "81914d34a8101da941cc3112f29a1eef"
server_key_iv: "a2899b5ca1e348e4cf95b7d1c73088fb"
server_key_version: "encryptionkey@0"
I20260808 07:01:22.908454 23129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 07:01:22.909041 23154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:22.909183 23129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:22.909262 23129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "01fdb6de77974963a35477a7fe852281"
format_stamp: "Formatted at 2026-08-08 07:01:22 on dist-test-slave-nj21"
server_key: "81914d34a8101da941cc3112f29a1eef"
server_key_iv: "a2899b5ca1e348e4cf95b7d1c73088fb"
server_key_version: "encryptionkey@0"
I20260808 07:01:22.909318 23129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:22.912364 23129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:22.914214 23129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:22.914335 23129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:22.918224 23129 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:33851
I20260808 07:01:22.918254 23218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:33851 every 8 connection(s)
I20260808 07:01:22.918522 23129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:22.919384 23222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:22.919814 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23129
I20260808 07:01:22.919919 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:22.920085 21617 external_mini_cluster.cc:1428] Setting key abbb671e823a37836be61b38d8b034c5
I20260808 07:01:22.922039 23222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281: Bootstrap starting.
I20260808 07:01:22.922187 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:22.920556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39501 (local address 127.21.28.126:33851)
0808 07:01:22.920705 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:22.920713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:22.920738 (+ 25us) server_negotiation.cc:407] Connection header received
0808 07:01:22.921113 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:22.921119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:22.921141 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:22.921244 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:22.921800 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:22.921803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:22.921803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:22.921806 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:22.921838 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:22.921848 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:22.921900 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:22.921980 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 07:01:22.922010 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260808 07:01:22.922700 23222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:22.923019 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:22.923009 23222 log.cc:824] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:22.923696 23222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281: No bootstrap required, opened a new log
I20260808 07:01:22.925315 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:22.924524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38573 (local address 127.21.28.126:33851)
0808 07:01:22.924560 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:22.924562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:22.924611 (+ 49us) server_negotiation.cc:407] Connection header received
0808 07:01:22.924677 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:22.924678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:22.924681 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:22.924721 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:22.925154 (+ 433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:22.925155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:22.925155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 07:01:22.925158 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:22.925167 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 07:01:22.925170 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:22.925208 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:22.925258 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 07:01:22.925267 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:01:22.925973 23222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fdb6de77974963a35477a7fe852281" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33851 } }
I20260808 07:01:22.926093 23222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 07:01:22.926124 23222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01fdb6de77974963a35477a7fe852281, State: Initialized, Role: FOLLOWER
I20260808 07:01:22.926235 23222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fdb6de77974963a35477a7fe852281" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33851 } }
I20260808 07:01:22.926303 23222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 07:01:22.926334 23222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 07:01:22.926378 23222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:22.926849 23222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fdb6de77974963a35477a7fe852281" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33851 } }
I20260808 07:01:22.926949 23222 leader_election.cc:302] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01fdb6de77974963a35477a7fe852281; no voters:
I20260808 07:01:22.927088 23222 leader_election.cc:288] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 07:01:22.927120 23234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:22.927260 23234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [term 1 LEADER]: Becoming Leader. State: Replica: 01fdb6de77974963a35477a7fe852281, State: Running, Role: LEADER
I20260808 07:01:22.927312 23222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:22.927346 23234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fdb6de77974963a35477a7fe852281" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33851 } }
I20260808 07:01:22.927657 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01fdb6de77974963a35477a7fe852281. Latest consensus state: current_term: 1 leader_uuid: "01fdb6de77974963a35477a7fe852281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fdb6de77974963a35477a7fe852281" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33851 } } }
I20260808 07:01:22.927655 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01fdb6de77974963a35477a7fe852281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fdb6de77974963a35477a7fe852281" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 33851 } } }
I20260808 07:01:22.927806 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:22.927867 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281 [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:22.928246 23245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:22.928654 23245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:22.930053 23245 catalog_manager.cc:1347] Generated new cluster ID: 80e67ca9bc4c482d9cecb445c6a19671
I20260808 07:01:22.930105 23245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:22.951038 23245 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:22.951190 23245 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:22.959733 23245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01fdb6de77974963a35477a7fe852281: Generated new TSK 0
I20260808 07:01:22.959894 23245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 07:01:22.962450 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23129
2026-08-08T07:01:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1372 ms)
[----------] 18 tests from AuthTokenIssuingTest (25011 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:23 dist-test-slave-nj21 krb5kdc[23265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:23 dist-test-slave-nj21 krb5kdc[23265](info): set up 2 sockets
Aug 08 07:01:23 dist-test-slave-nj21 krb5kdc[23265](info): commencing operation
krb5kdc: starting...
W20260808 07:01:24.079921 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:01:24.187621 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36811
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36811,127.21.28.125:46753,127.21.28.124:34199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:24.265126 23281 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:01:24.265314 23281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:24.265331 23281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:24.265349 23281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:24.266453 23281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:24.266490 23281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:24.266502 23281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:24.266515 23281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:24.266537 23281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:24.268034 23281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36811,127.21.28.125:46753,127.21.28.124:34199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:36811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:24.268249 23281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:24.268451 23281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:24.272864 23294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:24.272897 23290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:24.273032 23281 server_base.cc:1034] running on GCE node
W20260808 07:01:24.272897 23292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:24.273300 23281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:24.273550 23281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:24.274685 23281 hybrid_clock.cc:648] HybridClock initialized: now 1786172484274673 us; error 30 us; skew 500 ppm
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:24.276989 23281 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:01:24.277383 23281 webserver.cc:460] Webserver started at http://127.21.28.126:42993/ using document root <none> and password file <none>
I20260808 07:01:24.277588 23281 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:24.277633 23281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:24.277750 23281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:24.278682 23281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "9445962123f04b30952936dc0a2e068c"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "a1b91059143a14b66e4ef4abd003d91b"
server_key_iv: "967cceb453396cd416272c1b21c53f1d"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.278986 23281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "9445962123f04b30952936dc0a2e068c"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "a1b91059143a14b66e4ef4abd003d91b"
server_key_iv: "967cceb453396cd416272c1b21c53f1d"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.280221 23281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 07:01:24.280841 23306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:24.280973 23281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:24.281055 23281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "9445962123f04b30952936dc0a2e068c"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "a1b91059143a14b66e4ef4abd003d91b"
server_key_iv: "967cceb453396cd416272c1b21c53f1d"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.281112 23281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:24.295821 23281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:24.297319 23281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:24.297439 23281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:24.301036 23281 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:36811
I20260808 07:01:24.301079 23378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:36811 every 8 connection(s)
I20260808 07:01:24.301352 23281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:24.301502 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23281
I20260808 07:01:24.301587 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:24.301760 21617 external_mini_cluster.cc:1428] Setting key 8b933a733e103e9c4464de81fa29f331
I20260808 07:01:24.302268 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:24.303946 23381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.28.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.125@KRBTEST.COM" created.
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:01:24.315431 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.304603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36811 (local address 127.0.0.1:48178)
0808 07:01:24.304746 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.304781 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.305198 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.305556 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.305559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.305782 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.305933 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.305937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.307324 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.307324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.307998 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.308001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.308623 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.308624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.308672 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.309074 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.309082 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.312495 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.313983 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.313989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.313991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.314175 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.314431 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.314431 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.314432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.314500 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.314883 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.314887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.314938 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.315011 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.315156 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260808 07:01:24.315440 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.304629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.21.28.126:36811)
0808 07:01:24.304772 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.304780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.304849 (+ 69us) server_negotiation.cc:407] Connection header received
0808 07:01:24.305378 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.305381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.305424 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.305524 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.306019 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.307241 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.308070 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.308535 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.308650 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.312572 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.312578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.312579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.312585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.313914 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.314259 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.314262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.314263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.314362 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.314550 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.314550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.314551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.314822 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.314904 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.315064 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.315228 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260808 07:01:24.316144 23381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
I20260808 07:01:24.316438 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.316333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42918)
0808 07:01:24.316371 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.316420 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 07:01:24.316550 23310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
I20260808 07:01:24.316907 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.316827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42928)
0808 07:01:24.316867 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.316889 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 07:01:24.316998 23381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:46753: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab.
Entry for principal kudu/127.21.28.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab.
I20260808 07:01:24.324255 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.125:46753
--webserver_interface=127.21.28.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab
--principal=kudu/127.21.28.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36811,127.21.28.125:46753,127.21.28.124:34199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 07:01:24.366196 23381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } attempt: 1
I20260808 07:01:24.366616 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.366506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42934)
0808 07:01:24.366556 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.366587 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 07:01:24.366917 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.366825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42938)
0808 07:01:24.366862 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.366885 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260808 07:01:24.367002 23381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:46753: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
W20260808 07:01:24.402784 23390 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:01:24.402982 23390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:24.403008 23390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:24.403030 23390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:24.404136 23390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:24.404171 23390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:24.404189 23390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:24.404206 23390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:24.404219 23390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:24.405654 23390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36811,127.21.28.125:46753,127.21.28.124:34199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab
--principal=kudu/127.21.28.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.125:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.28.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:24.405897 23390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:24.406092 23390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:24.410508 23400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:24.410493 23397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:24.410517 23390 server_base.cc:1034] running on GCE node
W20260808 07:01:24.410491 23398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:24.410902 23390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:24.411165 23390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:24.412293 23390 hybrid_clock.cc:648] HybridClock initialized: now 1786172484412283 us; error 27 us; skew 500 ppm
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:24.414569 23390 init.cc:377] Logged in from keytab as kudu/127.21.28.125@KRBTEST.COM (short username kudu)
I20260808 07:01:24.414927 23390 webserver.cc:460] Webserver started at http://127.21.28.125:43263/ using document root <none> and password file <none>
I20260808 07:01:24.415094 23390 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:24.415130 23390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:24.415215 23390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:24.416162 23390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/data/instance:
uuid: "2866bf9914174bfa97b9ae28f8638721"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "6d3178b670dcc403803abac67d3dc9b0"
server_key_iv: "709a56828126c970456a22ce014b190b"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.416438 23390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/wal/instance:
uuid: "2866bf9914174bfa97b9ae28f8638721"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "6d3178b670dcc403803abac67d3dc9b0"
server_key_iv: "709a56828126c970456a22ce014b190b"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.417537 23390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:01:24.418157 23409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:24.418282 23390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:24.418358 23390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/wal
uuid: "2866bf9914174bfa97b9ae28f8638721"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "6d3178b670dcc403803abac67d3dc9b0"
server_key_iv: "709a56828126c970456a22ce014b190b"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.418423 23390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:24.431484 23390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:24.433358 23390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:24.433482 23390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:24.435158 23381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } attempt: 2
I20260808 07:01:24.435568 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.435423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42942)
0808 07:01:24.435486 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.435527 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 07:01:24.435941 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.435828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42950)
0808 07:01:24.435879 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.435912 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 07:01:24.436054 23381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:46753: Network error: Client connection negotiation failed: client connection to 127.21.28.125:46753: connect: Connection refused (error 111)
I20260808 07:01:24.437237 23390 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.125:46753
I20260808 07:01:24.437269 23489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.125:46753 every 8 connection(s)
I20260808 07:01:24.437539 23390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
I20260808 07:01:24.437943 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23390
I20260808 07:01:24.438012 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-1/wal/instance
I20260808 07:01:24.438182 21617 external_mini_cluster.cc:1428] Setting key 471b529c5af6ee29aa1090ec5717e39a
I20260808 07:01:24.438477 23490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:24.439920 23490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.28.124@KRBTEST.COM; defaulting to no policy
I20260808 07:01:24.447921 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.440520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.21.28.126:36811)
0808 07:01:24.440561 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.440562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.440779 (+ 217us) server_negotiation.cc:407] Connection header received
0808 07:01:24.441225 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.441226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.441239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.441295 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.441880 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.442610 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.443512 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.443843 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.443924 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.445720 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.445722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.445722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.445729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.446766 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.447089 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.447090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.447090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.447134 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.447303 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.447303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.447303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.447394 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.447442 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.447779 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.447861 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 07:01:24.448017 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.440524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36811 (local address 127.0.0.1:48194)
0808 07:01:24.440672 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.440702 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.441138 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.441356 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.441360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.441581 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.441770 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.441775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.442689 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.442690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.443436 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.443437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.443924 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.443925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.444022 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.444417 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.444424 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.445639 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.446830 (+ 1191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.446836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.446837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.447028 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.447174 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.447174 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.447174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.447262 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.447446 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.447450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.447590 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.447714 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.447824 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.124@KRBTEST.COM" created.
I20260808 07:01:24.448585 23490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 07:01:24.455332 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.448787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42956)
0808 07:01:24.448825 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.448835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.448869 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.449139 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.449140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.449241 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.449283 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.449284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.450194 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.450195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.450926 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.450929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.451347 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.451348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.451417 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.451800 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.451805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.453036 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.454472 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.454473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.454473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.454620 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.454785 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.454786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.454786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.454838 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.455060 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.455060 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.455094 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.455154 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.455244 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260808 07:01:24.455420 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.448845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.21.28.125:46753)
0808 07:01:24.448984 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.448989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.449008 (+ 19us) server_negotiation.cc:407] Connection header received
0808 07:01:24.449029 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.449031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.449044 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.449125 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.449340 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.450121 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.450999 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.451281 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.451325 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.453107 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.453114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.453114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.453120 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.454402 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.454682 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.454685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.454685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.454734 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.454885 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.454886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.454886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.455002 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.455070 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.455188 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.455311 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260808 07:01:24.455920 23490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } has no permanent_uuid. Determining permanent_uuid...
I20260808 07:01:24.456198 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.456102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59284)
0808 07:01:24.456139 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.456178 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 07:01:24.456319 23414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
I20260808 07:01:24.456681 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.456610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59298)
0808 07:01:24.456643 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.456665 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260808 07:01:24.456764 23490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34199: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab.
Entry for principal kudu/127.21.28.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab.
I20260808 07:01:24.462114 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.124:34199
--webserver_interface=127.21.28.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab
--principal=kudu/127.21.28.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:36811,127.21.28.125:46753,127.21.28.124:34199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 07:01:24.505990 23490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } attempt: 1
I20260808 07:01:24.506450 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.506328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59310)
0808 07:01:24.506390 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.506420 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 07:01:24.506744 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.506667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59318)
0808 07:01:24.506702 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.506725 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260808 07:01:24.506830 23490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34199: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
I20260808 07:01:24.527202 23381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } attempt: 3
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 07:01:24.533420 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.527565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.21.28.125:46753)
0808 07:01:24.527621 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.527622 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.527631 (+ 9us) server_negotiation.cc:407] Connection header received
0808 07:01:24.527655 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.527656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.527686 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.527735 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.528030 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.528516 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.529182 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.529394 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.529462 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.531203 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.531205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.531205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.531211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.532462 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.532750 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.532751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.532752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.532803 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.532963 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.532964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.532964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.533069 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.533124 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.533259 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.533333 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 07:01:24.533430 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.527482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42966)
0808 07:01:24.527526 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.527538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.527576 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.527742 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.527743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.527893 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.527959 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.527961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.528589 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.528590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.529106 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.529107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.529470 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.529472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.529540 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.530006 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.530010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.531126 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.532534 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.532535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.532536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.532691 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.532857 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.532857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.532858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.532913 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.533124 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.533125 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.533171 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.533231 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.533325 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 07:01:24.533764 23381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } has no permanent_uuid. Determining permanent_uuid...
I20260808 07:01:24.534060 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.533971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59324)
0808 07:01:24.534009 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.534035 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 07:01:24.534377 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.534311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59340)
0808 07:01:24.534345 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.534365 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 07:01:24.534463 23381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34199: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
W20260808 07:01:24.541107 23499 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:01:24.541276 23499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:24.541292 23499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:24.541316 23499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:24.542358 23499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:24.542388 23499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:24.542407 23499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:24.542423 23499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:24.542434 23499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:24.543848 23499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:36811,127.21.28.125:46753,127.21.28.124:34199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab
--principal=kudu/127.21.28.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.124:34199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.28.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:24.544016 23499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:24.544181 23499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:24.548520 23509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:24.548540 23506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:24.548604 23499 server_base.cc:1034] running on GCE node
W20260808 07:01:24.548533 23507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:24.548986 23499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:24.549259 23499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:24.550400 23499 hybrid_clock.cc:648] HybridClock initialized: now 1786172484550382 us; error 28 us; skew 500 ppm
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:24.552688 23499 init.cc:377] Logged in from keytab as kudu/127.21.28.124@KRBTEST.COM (short username kudu)
I20260808 07:01:24.553110 23499 webserver.cc:460] Webserver started at http://127.21.28.124:39823/ using document root <none> and password file <none>
I20260808 07:01:24.553323 23499 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:24.553368 23499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:24.553491 23499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:24.554430 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/data/instance:
uuid: "ec736a668a5d4e078d446af5072f9ab5"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "d62aa394c8d2e61fa6a0712664d8ed56"
server_key_iv: "afc6f4188882942bfb98c61d12817453"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.554740 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/wal/instance:
uuid: "ec736a668a5d4e078d446af5072f9ab5"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "d62aa394c8d2e61fa6a0712664d8ed56"
server_key_iv: "afc6f4188882942bfb98c61d12817453"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.555946 23499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 07:01:24.556558 23516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:24.556702 23499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:24.556782 23499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/wal
uuid: "ec736a668a5d4e078d446af5072f9ab5"
format_stamp: "Formatted at 2026-08-08 07:01:24 on dist-test-slave-nj21"
server_key: "d62aa394c8d2e61fa6a0712664d8ed56"
server_key_iv: "afc6f4188882942bfb98c61d12817453"
server_key_version: "encryptionkey@0"
I20260808 07:01:24.556843 23499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:24.565585 23381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } attempt: 1
I20260808 07:01:24.565920 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.565824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59356)
0808 07:01:24.565870 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.565895 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 07:01:24.566184 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.566114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59368)
0808 07:01:24.566146 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.566169 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260808 07:01:24.566257 23381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34199: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
I20260808 07:01:24.571724 23499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:24.573278 23499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:24.573411 23499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:24.574958 23490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } attempt: 2
I20260808 07:01:24.575281 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.575174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59372)
0808 07:01:24.575220 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.575249 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 07:01:24.575655 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.575533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59388)
0808 07:01:24.575598 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.575628 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 07:01:24.575776 23490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:34199: Network error: Client connection negotiation failed: client connection to 127.21.28.124:34199: connect: Connection refused (error 111)
I20260808 07:01:24.577461 23499 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.124:34199
I20260808 07:01:24.577497 23585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.124:34199 every 8 connection(s)
I20260808 07:01:24.577787 23499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
I20260808 07:01:24.578648 23588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:24.580104 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:01:24.586164 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23499
I20260808 07:01:24.586241 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786172218768454-21617-0/minicluster-data/master-2/wal/instance
I20260808 07:01:24.586391 21617 external_mini_cluster.cc:1428] Setting key fc0089bee2f8cc358c8a5b0c4ef2c77c
I20260808 07:01:24.587658 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.580668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.21.28.126:36811)
0808 07:01:24.580707 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.580708 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.580888 (+ 180us) server_negotiation.cc:407] Connection header received
0808 07:01:24.581295 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.581296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.581305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.581360 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.581853 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.582296 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.583184 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.583371 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.583415 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.585277 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.585278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.585278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.585284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.586373 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.586772 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.586774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.586774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.586832 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.587082 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.587083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.587084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.587202 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.587252 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.587543 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.587603 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:01:24.587909 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.580638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:36811 (local address 127.0.0.1:48202)
0808 07:01:24.580789 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.580819 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.581228 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.581412 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.581415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.581626 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.581795 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.581798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.582363 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.582364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.583128 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.583130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.583430 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.583430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.583524 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.583953 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.583960 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.585204 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.586445 (+ 1241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.586452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.586454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.586710 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.586891 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.586892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.586892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.587022 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.587262 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.587267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.587415 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.587522 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.587692 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260808 07:01:24.588399 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 07:01:24.594430 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.586724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.21.28.124:34199)
0808 07:01:24.586857 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.586861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.586945 (+ 84us) server_negotiation.cc:407] Connection header received
0808 07:01:24.586994 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.586996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.587006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.587066 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.587765 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.588605 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.589463 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.589805 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.589853 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.592030 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.592037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.592037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.592043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.593380 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.593676 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.593679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.593679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.593743 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.593918 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.593919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.593919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.594048 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.594174 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.594253 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.594323 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260808 07:01:24.594563 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.588678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.21.28.125:46753)
0808 07:01:24.588714 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.588716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.588726 (+ 10us) server_negotiation.cc:407] Connection header received
0808 07:01:24.588750 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.588750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.588757 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.588797 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.589053 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.589515 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.590239 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.590466 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.590516 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.592418 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.592419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.592420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.592427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.593615 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.593930 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.593931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.593931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.593982 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.594130 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.594131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.594131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.594233 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.594285 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.594410 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.594476 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 07:01:24.594563 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.588624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:46753 (local address 127.0.0.1:42974)
0808 07:01:24.588668 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.588678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.588710 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.588801 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.588801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.588942 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.588991 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.588993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.589585 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.589585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.590166 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.590168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.590532 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.590533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.590602 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.590864 (+ 262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.590869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.592340 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.593688 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.593689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.593689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.593865 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.594031 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.594032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.594032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.594082 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.594282 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.594283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.594323 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.594380 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.594474 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 07:01:24.595192 23588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:01:24.601800 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.595395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59412)
0808 07:01:24.595431 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.595439 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.595471 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.595585 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.595586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.595740 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.595769 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.595770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.596492 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.596493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.597078 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.597079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.597956 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.597957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.598011 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.598395 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.598399 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.599610 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.600938 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.600939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.600940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.601108 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.601281 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.601282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.601282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.601325 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.601546 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.601547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.601576 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.601617 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.601687 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 07:01:24.601852 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.595438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.21.28.124:34199)
0808 07:01:24.595463 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.595465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.595471 (+ 6us) server_negotiation.cc:407] Connection header received
0808 07:01:24.595521 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.595522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.595530 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.595577 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.595836 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.596434 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.597141 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.597480 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.597554 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.599725 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.599726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.599726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.599732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.600871 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.601169 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.601170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.601170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.601222 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.601372 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.601373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.601373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.601489 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.601554 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.601642 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.601739 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":14}
I20260808 07:01:24.602632 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.595435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.21.28.126:36811)
0808 07:01:24.595464 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.595466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.595597 (+ 131us) server_negotiation.cc:407] Connection header received
0808 07:01:24.595641 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.595641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.595651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.595738 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.596220 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.596771 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.597563 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.597836 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.597905 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.600227 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.600228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.600228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.600234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.601632 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.601996 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.601997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.601997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.602048 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.602213 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.602214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.602214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.602323 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.602430 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.602522 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.602576 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 07:01:24.603204 23588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5: Bootstrap starting.
I20260808 07:01:24.604152 23588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:24.604444 23588 log.cc:824] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:24.605180 23588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5: No bootstrap required, opened a new log
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 07:01:24.607100 23588 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } }
I20260808 07:01:24.607209 23588 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec736a668a5d4e078d446af5072f9ab5, State: Initialized, Role: FOLLOWER
I20260808 07:01:24.607311 23588 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } }
I20260808 07:01:24.607605 23588 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:24.607565 23596 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.607664 23596 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 07:01:24.609301 23616 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:01:24.609408 23616 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:01:24.610155 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.603276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.21.28.125:46753)
0808 07:01:24.603309 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.603310 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.603320 (+ 10us) server_negotiation.cc:407] Connection header received
0808 07:01:24.603353 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.603354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.603362 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.603395 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.603935 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.604395 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.605180 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.605397 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.605443 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.607595 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.607596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.607596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.607602 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.609075 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.609486 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.609487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.609487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.609539 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.609707 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.609708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.609708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.609822 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.609953 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.610010 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.610073 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 07:01:24.611217 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:24.609858 23499 master_runner.cc:186] Error getting master registration for 127.21.28.126:36811: OK, instance_id { permanent_uuid: "9445962123f04b30952936dc0a2e068c" instance_seqno: 1786172484300056 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 07:01:24.612557 23499 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 07:01:24.612260 23346 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9445962123f04b30952936dc0a2e068c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48212
I20260808 07:01:24.641382 23381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } attempt: 2
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
I20260808 07:01:24.647953 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.641789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.21.28.124:34199)
0808 07:01:24.641848 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.641851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.641874 (+ 23us) server_negotiation.cc:407] Connection header received
0808 07:01:24.641898 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.641899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.641908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.641953 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.642215 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.642699 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.643329 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.643526 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.643579 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.645341 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.645342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.645342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.645348 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.646791 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.647164 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.647165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.647165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.647214 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.647386 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.647387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.647387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.647557 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.647617 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.647783 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.647857 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 07:01:24.647965 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.641710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59424)
0808 07:01:24.641768 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.641778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.641838 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.641957 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.641958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.642105 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.642150 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.642151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.642768 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.642769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.643259 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.643260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.643584 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.643584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.643638 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.644097 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.644101 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.645262 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.646872 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.646873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.646873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.647095 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.647269 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.647270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.647270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.647333 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.647615 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.647616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.647663 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.647760 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.647854 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 07:01:24.649581 23381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c: Bootstrap starting.
I20260808 07:01:24.650296 23381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:24.650614 23381 log.cc:824] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:24.651333 23381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c: No bootstrap required, opened a new log
I20260808 07:01:24.652383 23381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } }
I20260808 07:01:24.652498 23381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9445962123f04b30952936dc0a2e068c, State: Initialized, Role: FOLLOWER
I20260808 07:01:24.652557 23381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } }
I20260808 07:01:24.652860 23381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:24.652841 23620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.652922 23620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:24.654084 23346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:01:24.654318 23346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [term 0 FOLLOWER]: Advancing to term 1
W20260808 07:01:24.654485 23641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:01:24.654529 23641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:01:24.654979 23346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } }
I20260808 07:01:24.655189 23346 leader_election.cc:288] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [CANDIDATE]: Term 1 election: Requested vote from peers 2866bf9914174bfa97b9ae28f8638721 (127.21.28.125:46753), ec736a668a5d4e078d446af5072f9ab5 (127.21.28.124:34199)
I20260808 07:01:24.655447 23554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9445962123f04b30952936dc0a2e068c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ec736a668a5d4e078d446af5072f9ab5"
I20260808 07:01:24.655565 23554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:24.656469 23554 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9445962123f04b30952936dc0a2e068c in term 1.
I20260808 07:01:24.656704 23310 leader_election.cc:302] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9445962123f04b30952936dc0a2e068c, ec736a668a5d4e078d446af5072f9ab5; no voters:
I20260808 07:01:24.656906 23620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:24.657038 23620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [term 1 LEADER]: Becoming Leader. State: Replica: 9445962123f04b30952936dc0a2e068c, State: Running, Role: LEADER
I20260808 07:01:24.657111 23620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } }
I20260808 07:01:24.657473 23620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9445962123f04b30952936dc0a2e068c. Latest consensus state: current_term: 1 leader_uuid: "9445962123f04b30952936dc0a2e068c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.657536 23620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:24.658202 23281 master_runner.cc:186] Error getting master registration for 127.21.28.125:46753: OK, instance_id { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" instance_seqno: 1786172484436275 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 07:01:24.660527 23281 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 07:01:24.663794 23651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:24.664029 23651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:24.666898 23490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } attempt: 3
I20260808 07:01:24.668237 23554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [term 1 FOLLOWER]: Refusing update from remote peer 9445962123f04b30952936dc0a2e068c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:01:24.668535 23644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:24.670264 23596 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9445962123f04b30952936dc0a2e068c. Latest consensus state: current_term: 1 leader_uuid: "9445962123f04b30952936dc0a2e068c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.670357 23596 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:24.673733 23644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9445962123f04b30952936dc0a2e068c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.673838 23644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:24.674322 23596 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9445962123f04b30952936dc0a2e068c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.674407 23596 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:24.674655 23657 mvcc.cc:205] Tried to move back new op lower bound from 7316162497198567424 to 7316162497156104192. Current Snapshot: MvccSnapshot[applied={T|T < 7316162497198567424}]
I20260808 07:01:24.674793 23664 mvcc.cc:205] Tried to move back new op lower bound from 7316162497198567424 to 7316162497156104192. Current Snapshot: MvccSnapshot[applied={T|T < 7316162497198567424}]
I20260808 07:01:24.675006 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.664371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.21.28.124:34199)
0808 07:01:24.664405 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.664407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.665184 (+ 777us) server_negotiation.cc:407] Connection header received
0808 07:01:24.665269 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.665270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.665278 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.665305 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.665894 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.666626 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.668637 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.668940 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.668978 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.671359 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.671360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.671361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.671367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.673068 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.673626 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.673627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.673628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.673694 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.674077 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.674078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.674079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.674211 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.674290 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.674743 (+ 453us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.674915 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Aug 08 07:01:24 dist-test-slave-nj21 krb5kdc[23265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172484, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
I20260808 07:01:24.678750 23651 catalog_manager.cc:1347] Generated new cluster ID: fdd7a35c60a0401eb36df099f21aaa67
I20260808 07:01:24.678798 23651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:24.689226 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.664006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.21.28.126:36811)
0808 07:01:24.664042 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.664045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.664420 (+ 375us) server_negotiation.cc:407] Connection header received
0808 07:01:24.664567 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.664568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.664577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.664618 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.667375 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.668110 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.678714 (+ 10604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.679006 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.679046 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.683223 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.683224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.683225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.683231 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.686387 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.687285 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.687286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.687286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.687354 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.688075 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.688076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.688076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.688221 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.688305 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.689060 (+ 755us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.689136 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 07:01:24.689473 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.667116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:34199 (local address 127.0.0.1:59434)
0808 07:01:24.667185 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:24.667194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:24.667225 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:24.671040 (+ 3815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:24.671041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:24.671216 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:24.671256 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.671258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.675417 (+ 4159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.675418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.676263 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:24.676266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.676956 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.676957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:24.677024 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.677399 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:24.677404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:24.678801 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:24.688387 (+ 9586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.688388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.688389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.688637 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.688891 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:24.688892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:24.688892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:24.688951 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:24.689196 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:24.689196 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:24.689235 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 07:01:24.689287 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 07:01:24.689367 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 07:01:24.689834 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.667280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.21.28.124:34199)
0808 07:01:24.670836 (+ 3556us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.670837 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.670866 (+ 29us) server_negotiation.cc:407] Connection header received
0808 07:01:24.670884 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.670884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.670893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.671332 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.671386 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.675292 (+ 3906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.676365 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.676850 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.677168 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.678880 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.678881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.678882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.678887 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.688324 (+ 9437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.688776 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.688777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.688777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.688835 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.689011 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.689012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.689012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.689129 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.689168 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.689641 (+ 473us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.689729 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3535,"thread_start_us":77,"threads_started":1}
I20260808 07:01:24.689985 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:24.664220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.21.28.125:46753)
0808 07:01:24.664266 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.664268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.664647 (+ 379us) server_negotiation.cc:407] Connection header received
0808 07:01:24.664748 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.664749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.664760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.664788 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.666267 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.666995 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.679191 (+ 12196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:24.679483 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:24.679522 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:24.682948 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:24.682949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:24.682950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:24.682956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:24.685682 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.687495 (+ 1813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.687496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.687497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.687562 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:24.688365 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:24.688366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:24.688366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:24.688520 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:24.688737 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:24.689804 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0808 07:01:24.689886 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 07:01:24.691112 23490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721: Bootstrap starting.
I20260808 07:01:24.691787 23490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:24.692085 23490 log.cc:824] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:24.692866 23490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721: No bootstrap required, opened a new log
I20260808 07:01:24.694794 23490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } }
I20260808 07:01:24.694928 23490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2866bf9914174bfa97b9ae28f8638721, State: Initialized, Role: FOLLOWER
I20260808 07:01:24.694996 23490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } }
I20260808 07:01:24.695295 23490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:24.695266 23671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.695348 23671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:24.696640 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9445962123f04b30952936dc0a2e068c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2866bf9914174bfa97b9ae28f8638721"
I20260808 07:01:24.696895 23450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [term 0 FOLLOWER]: Advancing to term 1
W20260808 07:01:24.697068 23687 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:01:24.697105 23687 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:01:24.697690 23450 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9445962123f04b30952936dc0a2e068c in term 1.
I20260808 07:01:24.698287 23449 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [term 1 FOLLOWER]: Refusing update from remote peer 9445962123f04b30952936dc0a2e068c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:01:24.698601 23620 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:24.699824 23671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9445962123f04b30952936dc0a2e068c. Latest consensus state: current_term: 1 leader_uuid: "9445962123f04b30952936dc0a2e068c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.699909 23671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:24.699940 23671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9445962123f04b30952936dc0a2e068c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.699976 23671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:24.703745 23651 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:24.704452 23651 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:24.706089 23644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9445962123f04b30952936dc0a2e068c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9445962123f04b30952936dc0a2e068c" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 36811 } } peers { permanent_uuid: "2866bf9914174bfa97b9ae28f8638721" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 46753 } } peers { permanent_uuid: "ec736a668a5d4e078d446af5072f9ab5" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 34199 } } }
I20260808 07:01:24.706169 23644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:24.707773 23689 mvcc.cc:205] Tried to move back new op lower bound from 7316162497198567424 to 7316162497156104192. Current Snapshot: MvccSnapshot[applied={T|T < 7316162497198567424}]
I20260808 07:01:24.714000 23651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9445962123f04b30952936dc0a2e068c: Generated new TSK 0
I20260808 07:01:24.714161 23651 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 07:01:24.742193 23387 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.740772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.21.28.126:36811)
0808 07:01:24.740813 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.740816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.741553 (+ 737us) server_negotiation.cc:407] Connection header received
0808 07:01:24.741648 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.741649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.741660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.741687 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.742125 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52208: BlockingRecv error: recv got EOF from 127.0.0.1:52208 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 07:01:24.743140 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.741130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.21.28.125:46753)
0808 07:01:24.741170 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.741171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.742394 (+ 1223us) server_negotiation.cc:407] Connection header received
0808 07:01:24.742663 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.742664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.742674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.742701 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.743072 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56618: BlockingRecv error: recv got EOF from 127.0.0.1:56618 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 07:01:24.743709 23660 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.741428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.21.28.124:34199)
0808 07:01:24.741468 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.741470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.742526 (+ 1056us) server_negotiation.cc:407] Connection header received
0808 07:01:24.742588 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.742589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.742599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.742651 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.743633 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42672: BlockingRecv error: recv got EOF from 127.0.0.1:42672 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 07:01:24.745647 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.744945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.21.28.125:46753)
0808 07:01:24.744984 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.744985 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.745184 (+ 199us) server_negotiation.cc:407] Connection header received
0808 07:01:24.745283 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.745284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.745301 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.745327 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.745602 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56634: BlockingRecv error: recv got EOF from 127.0.0.1:56634 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 07:01:24.748509 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.747813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.21.28.125:46753)
0808 07:01:24.747847 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.747849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.748070 (+ 221us) server_negotiation.cc:407] Connection header received
0808 07:01:24.748155 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.748156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.748164 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.748190 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.748470 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56646: BlockingRecv error: recv got EOF from 127.0.0.1:56646 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 07:01:24.752342 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.751610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.21.28.125:46753)
0808 07:01:24.751643 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.751645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.751883 (+ 238us) server_negotiation.cc:407] Connection header received
0808 07:01:24.751982 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.751983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.751992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.752017 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.752303 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56656: BlockingRecv error: recv got EOF from 127.0.0.1:56656 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 07:01:24.758138 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.757396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.21.28.125:46753)
0808 07:01:24.757426 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.757428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.757662 (+ 234us) server_negotiation.cc:407] Connection header received
0808 07:01:24.757750 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.757751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.757760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.757785 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.758100 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56670: BlockingRecv error: recv got EOF from 127.0.0.1:56670 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 07:01:24.767966 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.767195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.21.28.125:46753)
0808 07:01:24.767229 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.767231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.767457 (+ 226us) server_negotiation.cc:407] Connection header received
0808 07:01:24.767545 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.767546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.767555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.767581 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.767922 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56672: BlockingRecv error: recv got EOF from 127.0.0.1:56672 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 07:01:24.785637 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.785041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.21.28.125:46753)
0808 07:01:24.785077 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.785079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.785181 (+ 102us) server_negotiation.cc:407] Connection header received
0808 07:01:24.785236 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.785237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.785247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.785294 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.785585 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56680: BlockingRecv error: recv got EOF from 127.0.0.1:56680 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 07:01:24.819159 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.818520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.21.28.125:46753)
0808 07:01:24.818561 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.818562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.818684 (+ 122us) server_negotiation.cc:407] Connection header received
0808 07:01:24.818738 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.818739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.818750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.818796 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.819100 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56692: BlockingRecv error: recv got EOF from 127.0.0.1:56692 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 07:01:24.884927 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:24.884218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.21.28.125:46753)
0808 07:01:24.884283 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:24.884285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:24.884362 (+ 77us) server_negotiation.cc:407] Connection header received
0808 07:01:24.884437 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:24.884439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:24.884450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:24.884502 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:24.884866 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56704: BlockingRecv error: recv got EOF from 127.0.0.1:56704 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 07:01:25.014740 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:25.014093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.21.28.125:46753)
0808 07:01:25.014169 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:25.014171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:25.014236 (+ 65us) server_negotiation.cc:407] Connection header received
0808 07:01:25.014304 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:25.014305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:25.014316 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:25.014367 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:25.014681 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56720: BlockingRecv error: recv got EOF from 127.0.0.1:56720 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 07:01:25.272519 23496 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:25.271935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.21.28.125:46753)
0808 07:01:25.271995 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:25.271997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:25.272026 (+ 29us) server_negotiation.cc:407] Connection header received
0808 07:01:25.272088 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:25.272089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:25.272099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:25.272146 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:25.272456 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56734: BlockingRecv error: recv got EOF from 127.0.0.1:56734 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 07:01:25.609840 23616 catalog_manager.cc:1259] Loaded cluster ID: fdd7a35c60a0401eb36df099f21aaa67
I20260808 07:01:25.609908 23616 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5: loading cluster ID for follower catalog manager: success
I20260808 07:01:25.610690 23616 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5: acquiring CA information for follower catalog manager: success
I20260808 07:01:25.610981 23616 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec736a668a5d4e078d446af5072f9ab5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 07:01:25.697496 23687 catalog_manager.cc:1259] Loaded cluster ID: fdd7a35c60a0401eb36df099f21aaa67
I20260808 07:01:25.697562 23687 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721: loading cluster ID for follower catalog manager: success
I20260808 07:01:25.698328 23687 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721: acquiring CA information for follower catalog manager: success
I20260808 07:01:25.698623 23687 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2866bf9914174bfa97b9ae28f8638721: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 07:01:25.787979 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:25.785708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.21.28.125:46753)
0808 07:01:25.785831 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:25.785833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:25.785844 (+ 11us) server_negotiation.cc:407] Connection header received
0808 07:01:25.785870 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:25.785871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:25.785927 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:25.786000 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:25.786121 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.786621 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.787316 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.787497 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.787537 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:25.787646 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:25.787790 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:25.787850 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:25.787871 (+ 21us) server_negotiation.cc:299] Negotiation successful
0808 07:01:25.787889 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260808 07:01:25.791666 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:25.789637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.21.28.124:34199)
0808 07:01:25.789755 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:25.789757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:25.789764 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:01:25.789798 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:25.789799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:25.789832 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:25.789894 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:25.789970 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.790432 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.791054 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.791232 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.791273 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:25.791384 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:25.791484 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:25.791540 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:25.791561 (+ 21us) server_negotiation.cc:299] Negotiation successful
0808 07:01:25.791580 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
I20260808 07:01:25.796308 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:25.793570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.21.28.125:46753)
0808 07:01:25.793619 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:25.793620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:25.793628 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:01:25.793651 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:25.793652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:25.793663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:25.793713 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:25.793884 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.794576 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.795546 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.795862 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.795929 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:25.796054 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:25.796083 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:25.796123 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:25.796211 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 07:01:25.796237 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 07:01:25.796542 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:25.793207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.21.28.126:36811)
0808 07:01:25.793302 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:25.793304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:25.793334 (+ 30us) server_negotiation.cc:407] Connection header received
0808 07:01:25.793381 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:25.793382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:25.793452 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:25.793530 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:25.793677 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.794449 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.795493 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.795763 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.795816 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:25.796004 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:25.796112 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:25.796167 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:25.796427 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 07:01:25.796457 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":51,"threads_started":1}
I20260808 07:01:25.797187 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:25.793553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.21.28.124:34199)
0808 07:01:25.793592 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:25.793593 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:25.793617 (+ 24us) server_negotiation.cc:407] Connection header received
0808 07:01:25.793711 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:25.793712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:25.793720 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:25.793762 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:25.794646 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.795157 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.796282 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:25.796777 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:25.796815 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:25.796983 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:25.797013 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:25.797053 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:25.797101 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 07:01:25.797120 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 07:01:25.798128 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23281
I20260808 07:01:25.806819 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23390
I20260808 07:01:25.816586 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23499
2026-08-08T07:01:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2858 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 07:01:25 dist-test-slave-nj21 krb5kdc[23810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 07:01:25 dist-test-slave-nj21 krb5kdc[23810](info): set up 2 sockets
Aug 08 07:01:25 dist-test-slave-nj21 krb5kdc[23810](info): commencing operation
krb5kdc: starting...
W20260808 07:01:26.951494 21617 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/test-user.keytab.
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T07:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T07:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.28.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
Entry for principal kudu/127.21.28.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab.
I20260808 07:01:27.060472 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:38013
--webserver_interface=127.21.28.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:38013,127.21.28.125:34137,127.21.28.124:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 07:01:27.138175 23830 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:01:27.138378 23830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:27.138394 23830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:27.138412 23830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:27.139472 23830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:27.139503 23830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:27.139515 23830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:27.139526 23830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:27.139541 23830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:27.140957 23830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:38013,127.21.28.125:34137,127.21.28.124:40645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.126.keytab
--principal=kudu/127.21.28.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.126:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.28.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:27.141162 23830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:27.141340 23830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:27.145792 23839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:27.145802 23836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:27.145887 23830 server_base.cc:1034] running on GCE node
W20260808 07:01:27.145812 23835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:27.146214 23830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:27.146489 23830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:27.147635 23830 hybrid_clock.cc:648] HybridClock initialized: now 1786172487147616 us; error 36 us; skew 500 ppm
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:27.154461 23830 init.cc:377] Logged in from keytab as kudu/127.21.28.126@KRBTEST.COM (short username kudu)
I20260808 07:01:27.154841 23830 webserver.cc:460] Webserver started at http://127.21.28.126:36389/ using document root <none> and password file <none>
I20260808 07:01:27.155025 23830 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:27.155066 23830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:27.155182 23830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:27.156188 23830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/data/instance:
uuid: "1ea9e5611d9443b6bd3e121cf039e9eb"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "2e0bb93304ec8467d2570031c6cb9e4e"
server_key_iv: "1dba209ec37bb3548e7a8d53c228a541"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.156507 23830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/wal/instance:
uuid: "1ea9e5611d9443b6bd3e121cf039e9eb"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "2e0bb93304ec8467d2570031c6cb9e4e"
server_key_iv: "1dba209ec37bb3548e7a8d53c228a541"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.157723 23830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 07:01:27.158332 23848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:27.158488 23830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:27.158581 23830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/wal
uuid: "1ea9e5611d9443b6bd3e121cf039e9eb"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "2e0bb93304ec8467d2570031c6cb9e4e"
server_key_iv: "1dba209ec37bb3548e7a8d53c228a541"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.158639 23830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:27.187363 23830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:27.188982 23830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:27.189086 23830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:27.192694 23830 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.126:38013
I20260808 07:01:27.192727 23924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.126:38013 every 8 connection(s)
I20260808 07:01:27.193053 23830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/data/info.pb
I20260808 07:01:27.193971 23925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:27.194339 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23830
I20260808 07:01:27.194424 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-0/wal/instance
I20260808 07:01:27.194595 21617 external_mini_cluster.cc:1428] Setting key 042193192ec6ae4df87d2a1bece1b464
I20260808 07:01:27.195535 23925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.28.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.125@KRBTEST.COM" created.
I20260808 07:01:27.205451 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.196193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.21.28.126:38013)
0808 07:01:27.196397 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.196402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.196461 (+ 59us) server_negotiation.cc:407] Connection header received
0808 07:01:27.196979 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.196982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.197001 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.197076 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.197761 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.198834 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.200091 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.200398 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.200472 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.202826 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.202834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.202834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.202841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.204135 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.204448 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.204450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.204450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.204536 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.204734 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.204735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.204735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.204925 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.204985 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.205112 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.205284 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260808 07:01:27.205454 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.196168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38013 (local address 127.0.0.1:53434)
0808 07:01:27.196330 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.196379 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.196841 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.197085 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.197087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.197433 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.197650 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.197657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.198930 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.198931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.200003 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.200006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.200499 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.200500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.200577 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.201002 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.201011 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.202746 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.204211 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.204217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.204219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.204391 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.204592 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.204593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.204593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.204687 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.204972 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.204975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.205022 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.205083 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.205239 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260808 07:01:27.206139 23925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } has no permanent_uuid. Determining permanent_uuid...
I20260808 07:01:27.206434 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.206328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36476)
0808 07:01:27.206364 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.206412 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 07:01:27.206552 23850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
I20260808 07:01:27.206910 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.206823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36492)
0808 07:01:27.206859 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.206881 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 07:01:27.207026 23925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:34137: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab.
Entry for principal kudu/127.21.28.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab.
I20260808 07:01:27.218197 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.125:34137
--webserver_interface=127.21.28.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab
--principal=kudu/127.21.28.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:38013,127.21.28.125:34137,127.21.28.124:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 07:01:27.256232 23925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } attempt: 1
I20260808 07:01:27.256726 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.256553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36496)
0808 07:01:27.256615 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.256689 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 07:01:27.257069 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.256983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36506)
0808 07:01:27.257023 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.257049 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 07:01:27.257165 23925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:34137: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
W20260808 07:01:27.295626 23936 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:01:27.295820 23936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:27.295836 23936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:27.295853 23936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:27.296900 23936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:27.296931 23936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:27.296944 23936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:27.296955 23936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:27.296968 23936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:27.298355 23936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:38013,127.21.28.125:34137,127.21.28.124:40645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.125.keytab
--principal=kudu/127.21.28.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.125:34137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.28.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:27.298538 23936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:27.298695 23936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:27.302947 23942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:27.303004 23943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:27.302958 23947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:27.303179 23936 server_base.cc:1034] running on GCE node
I20260808 07:01:27.303395 23936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:27.303648 23936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:27.304798 23936 hybrid_clock.cc:648] HybridClock initialized: now 1786172487304788 us; error 27 us; skew 500 ppm
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:27.307030 23936 init.cc:377] Logged in from keytab as kudu/127.21.28.125@KRBTEST.COM (short username kudu)
I20260808 07:01:27.307422 23936 webserver.cc:460] Webserver started at http://127.21.28.125:45679/ using document root <none> and password file <none>
I20260808 07:01:27.307615 23936 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:27.307657 23936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:27.307798 23936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:27.308740 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/data/instance:
uuid: "cd3c08479b55403e84fd287fc9a55fdb"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "31432371bee288f66b69fa1df1c41296"
server_key_iv: "94d9700c8a62af02679317395a63fb43"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.309043 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/wal/instance:
uuid: "cd3c08479b55403e84fd287fc9a55fdb"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "31432371bee288f66b69fa1df1c41296"
server_key_iv: "94d9700c8a62af02679317395a63fb43"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.310199 23936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 07:01:27.310823 23958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:27.310972 23936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 07:01:27.311049 23936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/wal
uuid: "cd3c08479b55403e84fd287fc9a55fdb"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "31432371bee288f66b69fa1df1c41296"
server_key_iv: "94d9700c8a62af02679317395a63fb43"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.311111 23936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:27.325284 23925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } attempt: 2
I20260808 07:01:27.325623 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.325496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36516)
0808 07:01:27.325551 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.325589 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 07:01:27.325889 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.325820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36528)
0808 07:01:27.325853 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.325873 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260808 07:01:27.325976 23925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.125:34137: Network error: Client connection negotiation failed: client connection to 127.21.28.125:34137: connect: Connection refused (error 111)
I20260808 07:01:27.334743 23936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:27.336326 23936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:27.336452 23936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:27.340126 23936 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.125:34137
I20260808 07:01:27.340157 24033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.125:34137 every 8 connection(s)
I20260808 07:01:27.340447 23936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/data/info.pb
I20260808 07:01:27.341303 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:27.342176 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 23936
I20260808 07:01:27.342283 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-1/wal/instance
I20260808 07:01:27.342512 21617 external_mini_cluster.cc:1428] Setting key 1b69095b94c8a2dc4143d037dbee38bc
I20260808 07:01:27.343132 24034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.28.124@KRBTEST.COM; defaulting to no policy
I20260808 07:01:27.351958 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.343803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.21.28.126:38013)
0808 07:01:27.343859 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.343861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.344091 (+ 230us) server_negotiation.cc:407] Connection header received
0808 07:01:27.344574 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.344576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.344588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.344655 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.345186 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.345680 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.347084 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.347397 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.347484 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.349416 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.349418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.349419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.349426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.350844 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.351171 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.351172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.351172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.351221 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.351389 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.351390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.351390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.351494 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.351551 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.351835 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.351898 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 07:01:27.352103 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.343787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38013 (local address 127.0.0.1:53450)
0808 07:01:27.343959 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.344004 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.344489 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.344702 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.344705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.344926 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.345102 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.345107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.345767 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.345768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.347009 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.347012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.347499 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.347500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.347609 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.348033 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.348040 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.349323 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.350921 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.350927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.350929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.351120 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.351262 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.351263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.351263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.351349 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.351555 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.351562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.351732 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.351816 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.351935 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260808 07:01:27.352573 24034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.28.124@KRBTEST.COM" created.
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 07:01:27.358860 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.352804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36532)
0808 07:01:27.352842 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.352852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.352884 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.353126 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.353127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.353262 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.353302 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.353304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.354072 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.354073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.354537 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.354538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.354941 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.354942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.354998 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.355318 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.355322 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.356348 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.357971 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.357972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.357972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.358115 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.358281 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.358281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.358281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.358333 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.358584 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.358585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.358619 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.358679 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.358769 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 07:01:27.358963 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.352855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.21.28.125:34137)
0808 07:01:27.352993 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.352996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.353017 (+ 21us) server_negotiation.cc:407] Connection header received
0808 07:01:27.353036 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.353038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.353050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.353108 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.353362 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.353994 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.354598 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.354880 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.354937 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.356421 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.356428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.356429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.356434 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.357902 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.358176 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.358179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.358179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.358231 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.358389 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.358389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.358390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.358526 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.358594 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.358704 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.358838 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260808 07:01:27.359411 24034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } has no permanent_uuid. Determining permanent_uuid...
I20260808 07:01:27.359717 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.359597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55840)
0808 07:01:27.359633 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.359693 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260808 07:01:27.359854 23961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
I20260808 07:01:27.360196 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.360122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55846)
0808 07:01:27.360158 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.360180 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 07:01:27.360294 24034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:40645: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.28.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab.
Entry for principal kudu/127.21.28.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab.
I20260808 07:01:27.366003 21617 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDMxMqG/build/release/bin/kudu
/tmp/dist-test-taskDMxMqG/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.124:40645
--webserver_interface=127.21.28.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab
--principal=kudu/127.21.28.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.28.84:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.28.126:38013,127.21.28.125:34137,127.21.28.124:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 07:01:27.409497 24034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } attempt: 1
I20260808 07:01:27.409934 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.409789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55850)
0808 07:01:27.409852 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.409896 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 07:01:27.410300 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.410211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55860)
0808 07:01:27.410252 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.410277 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 07:01:27.410403 24034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:40645: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
I20260808 07:01:27.417114 23925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } attempt: 3
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 07:01:27.423377 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.417447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.21.28.125:34137)
0808 07:01:27.417499 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.417501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.417509 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:01:27.417532 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.417533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.417543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.417587 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.417877 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.418352 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.419017 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.419243 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.419297 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.421245 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.421246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.421246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.421252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.422447 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.422799 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.422800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.422800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.422843 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.423001 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.423001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.423001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.423093 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.423135 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.423268 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.423323 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 07:01:27.423452 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.417367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36534)
0808 07:01:27.417418 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.417429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.417467 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.417600 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.417600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.417747 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.417809 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.417811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.418420 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.418421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.418947 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.418948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.419303 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.419305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.419377 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.419785 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.419789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.421170 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.422523 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.422524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.422524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.422738 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.422890 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.422891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.422891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.422952 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.423139 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.423139 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.423188 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.423250 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.423346 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 07:01:27.423848 23925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } has no permanent_uuid. Determining permanent_uuid...
I20260808 07:01:27.424149 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.424053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55872)
0808 07:01:27.424096 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.424126 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 07:01:27.424436 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.424357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55882)
0808 07:01:27.424391 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.424420 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 07:01:27.424505 23925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:40645: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
W20260808 07:01:27.444957 24049 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 07:01:27.445128 24049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 07:01:27.445145 24049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 07:01:27.445178 24049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 07:01:27.446251 24049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 07:01:27.446286 24049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 07:01:27.446300 24049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 07:01:27.446321 24049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 07:01:27.446343 24049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 07:01:27.447768 24049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.28.84:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.28.126:38013,127.21.28.125:34137,127.21.28.124:40645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/krb5kdc/kudu_127.21.28.124.keytab
--principal=kudu/127.21.28.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.28.124:40645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.28.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 05:11:30 UTC on ccycloud.quasar-iqclcm.root.comops.site
build id 4574014
I20260808 07:01:27.447983 24049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 07:01:27.448204 24049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 07:01:27.452724 24062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:27.452750 24057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 07:01:27.452775 24059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 07:01:27.452912 24049 server_base.cc:1034] running on GCE node
I20260808 07:01:27.453207 24049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 07:01:27.453469 24049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 07:01:27.454609 24049 hybrid_clock.cc:648] HybridClock initialized: now 1786172487454598 us; error 26 us; skew 500 ppm
I20260808 07:01:27.455632 23925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } attempt: 1
I20260808 07:01:27.455984 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.455875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55890)
0808 07:01:27.455925 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.455957 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 07:01:27.456264 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.456182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55900)
0808 07:01:27.456218 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.456244 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 07:01:27.456344 23925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.28.124:40645: Network error: Client connection negotiation failed: client connection to 127.21.28.124:40645: connect: Connection refused (error 111)
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 07:01:27.457144 24049 init.cc:377] Logged in from keytab as kudu/127.21.28.124@KRBTEST.COM (short username kudu)
I20260808 07:01:27.457537 24049 webserver.cc:460] Webserver started at http://127.21.28.124:46797/ using document root <none> and password file <none>
I20260808 07:01:27.457700 24049 fs_manager.cc:359] Metadata directory not provided
I20260808 07:01:27.457736 24049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 07:01:27.457821 24049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 07:01:27.458714 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/data/instance:
uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "d26c321c781f9ccb7f86f901e72626f7"
server_key_iv: "29ffb5f65315358fa71e417759a76e72"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.458994 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/wal/instance:
uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "d26c321c781f9ccb7f86f901e72626f7"
server_key_iv: "29ffb5f65315358fa71e417759a76e72"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.460229 24049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 07:01:27.460841 24071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:27.461025 24049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 07:01:27.461113 24049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/data,/tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/wal
uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f"
format_stamp: "Formatted at 2026-08-08 07:01:27 on dist-test-slave-nj21"
server_key: "d26c321c781f9ccb7f86f901e72626f7"
server_key_iv: "29ffb5f65315358fa71e417759a76e72"
server_key_version: "encryptionkey@0"
I20260808 07:01:27.461179 24049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 07:01:27.473387 24049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 07:01:27.474879 24049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 07:01:27.475006 24049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 07:01:27.478547 24034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } attempt: 2
I20260808 07:01:27.478638 24049 rpc_server.cc:307] RPC server started. Bound to: 127.21.28.124:40645
I20260808 07:01:27.478680 24136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.28.124:40645 every 8 connection(s)
I20260808 07:01:27.479012 24049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/data/info.pb
I20260808 07:01:27.479753 24137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 07:01:27.479929 21617 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDMxMqG/build/release/bin/kudu as pid 24049
I20260808 07:01:27.480039 21617 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDMxMqG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786172218768454-21617-0/minicluster-data/master-2/wal/instance
I20260808 07:01:27.480252 21617 external_mini_cluster.cc:1428] Setting key f84618365235b6e155acd32bcd0c0cdd
I20260808 07:01:27.481828 24137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.125@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:01:27.488171 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.478833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55908)
0808 07:01:27.478880 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.478892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.478944 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.480277 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.480278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.480443 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.480499 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.480502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.481884 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.481886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.482597 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.482600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.483335 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.483337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.483398 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.483840 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.483846 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.485106 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.486909 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.486910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.486911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.487092 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.487302 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.487303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.487303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.487364 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.487839 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.487841 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.487887 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.487978 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.488069 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 07:01:27.488344 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.478982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.21.28.124:40645)
0808 07:01:27.479554 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.479562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.479585 (+ 23us) server_negotiation.cc:407] Connection header received
0808 07:01:27.480081 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.480090 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.480149 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.480276 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.480774 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.481793 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.482684 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.483262 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.483358 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.485217 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.485222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.485222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.485233 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.486840 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.487160 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.487163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.487163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.487246 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.487440 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.487441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.487442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.487743 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.487846 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.488022 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.488157 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":523,"thread_start_us":488,"threads_started":1}
I20260808 07:01:27.489334 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.480621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.21.28.124:40645)
0808 07:01:27.480772 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.480773 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.480824 (+ 51us) server_negotiation.cc:407] Connection header received
0808 07:01:27.480886 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.480887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.480896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.480974 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.481718 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.482293 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.483311 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.483579 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.483632 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.486157 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.486158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.486158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.486165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.488238 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.488591 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.488592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.488593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.488646 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.488909 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.488910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.488910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.489008 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.489099 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.489216 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.489270 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260808 07:01:27.490003 24034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb: Bootstrap starting.
I20260808 07:01:27.490861 24034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:27.491122 24034 log.cc:824] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:27.491874 24034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb: No bootstrap required, opened a new log
I20260808 07:01:27.493152 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.482487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.21.28.126:38013)
0808 07:01:27.482528 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.482529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.482678 (+ 149us) server_negotiation.cc:407] Connection header received
0808 07:01:27.482761 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.482762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.482773 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.482818 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.483206 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.483831 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.484909 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.485190 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.485255 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.487449 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.487450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.487451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.487458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.491949 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.492350 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.492352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.492352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.492425 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.492617 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.492618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.492618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.492745 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.492811 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.492986 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.493064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:01:27.493172 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.482435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.126:38013 (local address 127.0.0.1:53458)
0808 07:01:27.482589 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.482627 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.482705 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.482836 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.482838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.483074 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.483129 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.483130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.483900 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.483901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.484746 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.484749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.485255 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.485255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.485351 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.485837 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.485842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.487384 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.492032 (+ 4648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.492038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.492040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.492279 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.492493 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.492494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.492494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.492557 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.492816 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.492823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.492870 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.492942 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.493042 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260808 07:01:27.493947 24137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } has no permanent_uuid. Determining permanent_uuid...
I20260808 07:01:27.494256 24034 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } }
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.126@KRBTEST.COM
I20260808 07:01:27.494356 24034 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c08479b55403e84fd287fc9a55fdb, State: Initialized, Role: FOLLOWER
I20260808 07:01:27.494446 24034 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } }
I20260808 07:01:27.494683 24148 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.494747 24034 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:27.494804 24148 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [sys.catalog]: This master's current role is: FOLLOWER
W20260808 07:01:27.496457 24162 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:01:27.496512 24162 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:01:27.498021 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.489841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.21.28.126:38013)
0808 07:01:27.489979 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.489981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.489988 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:01:27.490005 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.490005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.490013 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.490073 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.490589 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.491381 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.492284 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.492579 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.492633 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.494782 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.494783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.494784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.494791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.496633 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.496984 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.496985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.496986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.497041 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.497220 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.497221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.497221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.497331 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.497470 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.497823 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.497898 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260808 07:01:27.496713 23936 master_runner.cc:186] Error getting master registration for 127.21.28.126:38013: OK, instance_id { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" instance_seqno: 1786172487191685 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 07:01:27.499293 23936 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
I20260808 07:01:27.502373 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.494150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.125:34137 (local address 127.0.0.1:36550)
0808 07:01:27.494794 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.495709 (+ 915us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.495784 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.495953 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.495954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.496096 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.496143 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.496145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.496932 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.496933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.497647 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.497649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.498136 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.498137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.498197 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.498521 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.498525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.500024 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.501414 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.501415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.501415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.501565 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.501725 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.501725 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.501726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.501778 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.502109 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.502110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.502149 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.502214 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.502294 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629}
I20260808 07:01:27.502641 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.494218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.21.28.125:34137)
0808 07:01:27.494259 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.494261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.495758 (+ 1497us) server_negotiation.cc:407] Connection header received
0808 07:01:27.495854 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.495855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.495866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.495892 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.496227 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.496841 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.497750 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.498034 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.498069 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.500091 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.500093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.500093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.500099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.501344 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.501623 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.501624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.501624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.501674 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.501862 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.501863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.501864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.502039 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.502111 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.502246 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.502557 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 07:01:27.503048 24137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.28.125@KRBTEST.COM
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.124@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
I20260808 07:01:27.508203 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.498746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.21.28.125:34137)
0808 07:01:27.499465 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.499467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.499475 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:01:27.499491 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.499492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.499501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.499597 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.500042 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.500719 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.501791 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.502059 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.502325 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.504589 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.504590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.504590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.504596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.506724 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.507108 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.507109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.507109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.507176 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.507365 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.507366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.507366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.507490 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.507978 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.508017 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.508090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":79,"threads_started":1}
I20260808 07:01:27.509101 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.503265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55928)
0808 07:01:27.503301 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.503309 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.503337 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.503691 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.503692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.503805 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.503825 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.503826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.504439 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.504439 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.504873 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.504874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.505254 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.505254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.505301 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.505605 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.505609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.506708 (+ 1099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.508051 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.508052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.508052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.508195 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.508467 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.508468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.508468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.508519 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.508784 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.508785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.508828 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.508893 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.508995 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 07:01:27.509172 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.503541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.21.28.124:40645)
0808 07:01:27.503588 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.503589 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.503596 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:01:27.503611 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.503612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.503619 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.503662 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.503882 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.504366 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.504934 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.505195 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.505249 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.506775 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.506776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.506777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.506782 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.507987 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.508344 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.508345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.508345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.508405 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.508572 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.508573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.508573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.508716 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.508774 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.508929 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.509048 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":31}
I20260808 07:01:27.509684 21617 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 07:01:27.510602 24137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f: Bootstrap starting.
I20260808 07:01:27.511149 24137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:27.511405 24137 log.cc:824] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:27.510613 23890 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53468
I20260808 07:01:27.512063 24137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f: No bootstrap required, opened a new log
I20260808 07:01:27.513625 24137 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } }
I20260808 07:01:27.513757 24137 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb63ddbe11e345d7ad37abfbfeb84a4f, State: Initialized, Role: FOLLOWER
I20260808 07:01:27.513839 24137 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } }
I20260808 07:01:27.514123 24137 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:27.514783 24171 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.514860 24171 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [sys.catalog]: This master's current role is: FOLLOWER
W20260808 07:01:27.516126 24186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 07:01:27.516191 24186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 07:01:27.516114 24049 master_runner.cc:186] Error getting master registration for 127.21.28.126:38013: OK, instance_id { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" instance_seqno: 1786172487191685 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 07:01:27.517657 24049 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 07:01:27.531469 23925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } attempt: 2
Aug 08 07:01:27 dist-test-slave-nj21 krb5kdc[23810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786172487, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.28.126@KRBTEST.COM for kudu/127.21.28.124@KRBTEST.COM
I20260808 07:01:27.538640 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.531768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.28.124:40645 (local address 127.0.0.1:55938)
0808 07:01:27.531817 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 07:01:27.531829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 07:01:27.531870 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 07:01:27.532033 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 07:01:27.532034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 07:01:27.532179 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 07:01:27.532220 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.532222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.533070 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.533071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.533551 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 07:01:27.533553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.533978 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.533979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 07:01:27.534032 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.534463 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 07:01:27.534468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 07:01:27.535871 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 07:01:27.537479 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.537480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.537480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.537704 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.537972 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 07:01:27.537973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 07:01:27.537973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 07:01:27.538068 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 07:01:27.538315 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 07:01:27.538328 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 07:01:27.538388 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 07:01:27.538448 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 07:01:27.538548 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 07:01:27.538637 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.531823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.21.28.124:40645)
0808 07:01:27.531858 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.531860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.531868 (+ 8us) server_negotiation.cc:407] Connection header received
0808 07:01:27.531948 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.531949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.531959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.532014 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.532291 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.532989 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.533626 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.533916 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.533978 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.535958 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.535960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.535960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.535967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.537394 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.537777 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.537778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.537778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.537853 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.538135 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.538136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.538136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.538266 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.538325 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.538478 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.538548 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 07:01:27.540760 23925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb: Bootstrap starting.
I20260808 07:01:27.541491 23925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb: Neither blocks nor log segments found. Creating new log.
I20260808 07:01:27.541801 23925 log.cc:824] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb: Log is configured to *not* fsync() on all Append() calls
I20260808 07:01:27.542588 23925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb: No bootstrap required, opened a new log
I20260808 07:01:27.543928 23925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } }
I20260808 07:01:27.544061 23925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea9e5611d9443b6bd3e121cf039e9eb, State: Initialized, Role: FOLLOWER
I20260808 07:01:27.544140 23925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } }
I20260808 07:01:27.544440 23925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 07:01:27.544479 23890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 07:01:27.544521 23890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:27.545308 23890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } }
I20260808 07:01:27.545538 23890 leader_election.cc:288] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [CANDIDATE]: Term 1 election: Requested vote from peers cd3c08479b55403e84fd287fc9a55fdb (127.21.28.125:34137), bb63ddbe11e345d7ad37abfbfeb84a4f (127.21.28.124:40645)
I20260808 07:01:27.546165 23997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd3c08479b55403e84fd287fc9a55fdb"
I20260808 07:01:27.546296 23997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:27.547236 23997 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea9e5611d9443b6bd3e121cf039e9eb in term 1.
I20260808 07:01:27.547747 23850 leader_election.cc:302] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea9e5611d9443b6bd3e121cf039e9eb, cd3c08479b55403e84fd287fc9a55fdb; no voters:
I20260808 07:01:27.548189 24104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f"
I20260808 07:01:27.548336 24104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [term 0 FOLLOWER]: Advancing to term 1
I20260808 07:01:27.549216 24104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea9e5611d9443b6bd3e121cf039e9eb in term 1.
I20260808 07:01:27.544416 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.549630 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:27.549700 24189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 07:01:27.549760 24189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [term 1 LEADER]: Becoming Leader. State: Replica: 1ea9e5611d9443b6bd3e121cf039e9eb, State: Running, Role: LEADER
I20260808 07:01:27.549834 24189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } }
I20260808 07:01:27.550220 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea9e5611d9443b6bd3e121cf039e9eb. Latest consensus state: current_term: 1 leader_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.550290 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:27.550392 23830 master_runner.cc:263] No leader master found from master 1ea9e5611d9443b6bd3e121cf039e9eb
I20260808 07:01:27.550431 23830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 07:01:27.550474 24220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 07:01:27.550925 24220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 07:01:27.563527 24104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [term 1 FOLLOWER]: Refusing update from remote peer 1ea9e5611d9443b6bd3e121cf039e9eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:01:27.563838 24189 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:27.565047 24171 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea9e5611d9443b6bd3e121cf039e9eb. Latest consensus state: current_term: 1 leader_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.565135 24171 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:27.565485 23997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [term 1 FOLLOWER]: Refusing update from remote peer 1ea9e5611d9443b6bd3e121cf039e9eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 07:01:27.565735 24189 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 07:01:27.566846 24148 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea9e5611d9443b6bd3e121cf039e9eb. Latest consensus state: current_term: 1 leader_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.566931 24148 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:27.568209 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.546824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.21.28.126:38013)
0808 07:01:27.546858 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.546860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.551688 (+ 4828us) server_negotiation.cc:407] Connection header received
0808 07:01:27.551802 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.551803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.551813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.551839 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.552568 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.553289 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.554305 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.554629 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.554670 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.559158 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.559159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.559159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.559166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.560688 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.567316 (+ 6628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.567318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.567318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.567376 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.567565 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.567566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.567566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.567744 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.567828 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.568032 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.568117 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 07:01:27.572587 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.572667 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:27.572707 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.572751 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb [sys.catalog]: This master's current role is: LEADER
I20260808 07:01:27.572912 24171 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.572997 24171 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:27.573053 24148 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea9e5611d9443b6bd3e121cf039e9eb" member_type: VOTER last_known_addr { host: "127.21.28.126" port: 38013 } } peers { permanent_uuid: "cd3c08479b55403e84fd287fc9a55fdb" member_type: VOTER last_known_addr { host: "127.21.28.125" port: 34137 } } peers { permanent_uuid: "bb63ddbe11e345d7ad37abfbfeb84a4f" member_type: VOTER last_known_addr { host: "127.21.28.124" port: 40645 } } }
I20260808 07:01:27.573123 24148 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 07:01:27.573206 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.547414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.21.28.124:40645)
0808 07:01:27.547447 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.547449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.555066 (+ 7617us) server_negotiation.cc:407] Connection header received
0808 07:01:27.555158 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.555159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.555168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.555197 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.555930 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.556623 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.557608 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.557899 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.557937 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.561615 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.561617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.561617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.561624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.563106 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.572170 (+ 9064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.572171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.572171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.572233 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.572427 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.572428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.572428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.572543 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.572626 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.572827 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.572903 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 07:01:27.573375 24226 mvcc.cc:205] Tried to move back new op lower bound from 7316162509053751296 to 7316162509004697600. Current Snapshot: MvccSnapshot[applied={T|T < 7316162509053751296}]
I20260808 07:01:27.573302 24225 mvcc.cc:205] Tried to move back new op lower bound from 7316162509053751296 to 7316162509004697600. Current Snapshot: MvccSnapshot[applied={T|T < 7316162509053751296}]
I20260808 07:01:27.573820 24229 mvcc.cc:205] Tried to move back new op lower bound from 7316162509053751296 to 7316162509004697600. Current Snapshot: MvccSnapshot[applied={T|T < 7316162509053751296}]
I20260808 07:01:27.579245 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:27.547437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.21.28.125:34137)
0808 07:01:27.547472 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.547474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.568890 (+ 21416us) server_negotiation.cc:407] Connection header received
0808 07:01:27.568983 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.568984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.568993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.569020 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.569549 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.570294 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.571243 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:27.571563 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:27.571601 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:27.576594 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 07:01:27.576596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 07:01:27.576596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 07:01:27.576603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 07:01:27.578098 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.578461 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.578462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.578462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.578514 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 07:01:27.578692 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 07:01:27.578693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 07:01:27.578693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 07:01:27.578826 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 07:01:27.578892 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:27.579081 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 07:01:27.579159 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 07:01:27.584095 24220 catalog_manager.cc:1347] Generated new cluster ID: 510eebc112ba417daf0dc507cdbdba8e
I20260808 07:01:27.584144 24220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 07:01:27.614759 24220 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 07:01:27.615525 24220 catalog_manager.cc:1504] Loading token signing keys...
I20260808 07:01:27.623982 24220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ea9e5611d9443b6bd3e121cf039e9eb: Generated new TSK 0
I20260808 07:01:27.624161 24220 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 07:01:27.650288 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.648712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.21.28.126:38013)
0808 07:01:27.648754 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.648757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.649594 (+ 837us) server_negotiation.cc:407] Connection header received
0808 07:01:27.649722 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.649723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.649733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.649763 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.650225 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53494: BlockingRecv error: recv got EOF from 127.0.0.1:53494 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 07:01:27.651027 24144 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.649090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.21.28.124:40645)
0808 07:01:27.649129 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.649131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.650533 (+ 1402us) server_negotiation.cc:407] Connection header received
0808 07:01:27.650619 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.650620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.650630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.650657 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.650970 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55956: BlockingRecv error: recv got EOF from 127.0.0.1:55956 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 07:01:27.652004 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.651358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.21.28.125:34137)
0808 07:01:27.651395 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.651396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.651506 (+ 110us) server_negotiation.cc:407] Connection header received
0808 07:01:27.651587 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.651587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.651597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.651623 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.651948 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36564: BlockingRecv error: recv got EOF from 127.0.0.1:36564 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 07:01:27.654096 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.653380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.21.28.125:34137)
0808 07:01:27.653411 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.653412 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.653600 (+ 188us) server_negotiation.cc:407] Connection header received
0808 07:01:27.653695 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.653696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.653712 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.653739 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.654055 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36574: BlockingRecv error: recv got EOF from 127.0.0.1:36574 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 07:01:27.656978 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.656270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.21.28.125:34137)
0808 07:01:27.656301 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.656302 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.656502 (+ 200us) server_negotiation.cc:407] Connection header received
0808 07:01:27.656589 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.656589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.656598 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.656624 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.656940 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36590: BlockingRecv error: recv got EOF from 127.0.0.1:36590 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 07:01:27.660651 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.659995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.21.28.125:34137)
0808 07:01:27.660023 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.660025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.660194 (+ 169us) server_negotiation.cc:407] Connection header received
0808 07:01:27.660270 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.660271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.660279 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.660303 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.660598 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36600: BlockingRecv error: recv got EOF from 127.0.0.1:36600 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260808 07:01:27.666384 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.665655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.21.28.125:34137)
0808 07:01:27.665689 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.665690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.665880 (+ 190us) server_negotiation.cc:407] Connection header received
0808 07:01:27.665989 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.665990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.666000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.666028 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.666341 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36614: BlockingRecv error: recv got EOF from 127.0.0.1:36614 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 07:01:27.679291 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.678573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.21.28.125:34137)
0808 07:01:27.678605 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.678607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.678802 (+ 195us) server_negotiation.cc:407] Connection header received
0808 07:01:27.678892 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.678893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.678915 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.678941 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.679247 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36630: BlockingRecv error: recv got EOF from 127.0.0.1:36630 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 07:01:27.697563 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.696967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.21.28.125:34137)
0808 07:01:27.697009 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.697011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.697122 (+ 111us) server_negotiation.cc:407] Connection header received
0808 07:01:27.697173 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.697174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.697184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.697228 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.697510 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36640: BlockingRecv error: recv got EOF from 127.0.0.1:36640 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 07:01:27.731137 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.730543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.21.28.125:34137)
0808 07:01:27.730616 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.730618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.730670 (+ 52us) server_negotiation.cc:407] Connection header received
0808 07:01:27.730727 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.730728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.730738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.730786 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.731073 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36644: BlockingRecv error: recv got EOF from 127.0.0.1:36644 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 07:01:27.796921 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.796244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.21.28.125:34137)
0808 07:01:27.796295 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.796297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.796434 (+ 137us) server_negotiation.cc:407] Connection header received
0808 07:01:27.796501 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.796502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.796513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.796561 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.796856 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36646: BlockingRecv error: recv got EOF from 127.0.0.1:36646 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 07:01:27.926645 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:27.925941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.21.28.125:34137)
0808 07:01:27.925999 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:27.926002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:27.926127 (+ 125us) server_negotiation.cc:407] Connection header received
0808 07:01:27.926192 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:27.926192 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:27.926205 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:27.926251 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:27.926582 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36660: BlockingRecv error: recv got EOF from 127.0.0.1:36660 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260808 07:01:28.184401 24169 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 07:01:28.183773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.21.28.125:34137)
0808 07:01:28.183830 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:28.183832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:28.183856 (+ 24us) server_negotiation.cc:407] Connection header received
0808 07:01:28.183935 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:28.183936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:28.183947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:28.183994 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 07:01:28.184338 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36662: BlockingRecv error: recv got EOF from 127.0.0.1:36662 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 07:01:28.496910 24162 catalog_manager.cc:1259] Loaded cluster ID: 510eebc112ba417daf0dc507cdbdba8e
I20260808 07:01:28.496974 24162 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb: loading cluster ID for follower catalog manager: success
I20260808 07:01:28.497722 24162 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb: acquiring CA information for follower catalog manager: success
I20260808 07:01:28.497974 24162 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd3c08479b55403e84fd287fc9a55fdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 07:01:28.516541 24186 catalog_manager.cc:1259] Loaded cluster ID: 510eebc112ba417daf0dc507cdbdba8e
I20260808 07:01:28.516595 24186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f: loading cluster ID for follower catalog manager: success
I20260808 07:01:28.517339 24186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f: acquiring CA information for follower catalog manager: success
I20260808 07:01:28.517652 24186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb63ddbe11e345d7ad37abfbfeb84a4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 07:01:28.699920 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:28.697404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.21.28.125:34137)
0808 07:01:28.697537 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:28.697539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:28.697550 (+ 11us) server_negotiation.cc:407] Connection header received
0808 07:01:28.697630 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:28.697631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:28.697683 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:28.697748 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:28.697887 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.698383 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.699119 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.699313 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.699366 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:28.699526 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:28.699637 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:28.699734 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:28.699800 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 07:01:28.699831 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260808 07:01:28.703747 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:28.701522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.21.28.124:40645)
0808 07:01:28.701634 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:28.701636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:28.701643 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:01:28.701680 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:28.701681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:28.701714 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:28.701777 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:28.701875 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.702354 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.703035 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.703233 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.703284 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:28.703423 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:28.703519 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:28.703579 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:28.703621 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 07:01:28.703641 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":45,"threads_started":1}
I20260808 07:01:28.707401 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:28.705372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.21.28.124:40645)
0808 07:01:28.705411 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:28.705413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:28.705420 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:01:28.705440 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:28.705440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:28.705448 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:28.705500 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:28.705647 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.706131 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.706865 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.707038 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.707088 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:28.707204 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:28.707226 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:28.707265 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:28.707314 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 07:01:28.707332 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 07:01:28.708415 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:28.705370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.21.28.126:38013)
0808 07:01:28.705495 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:28.705498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:28.705505 (+ 7us) server_negotiation.cc:407] Connection header received
0808 07:01:28.705548 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:28.705548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:28.705602 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:28.705676 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:28.705776 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.706241 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.707246 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.707555 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.707643 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:28.707830 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:28.707936 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:28.708001 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:28.708299 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 07:01:28.708329 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260808 07:01:28.708678 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 07:01:28.705186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.21.28.125:34137)
0808 07:01:28.705318 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 07:01:28.705320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 07:01:28.705503 (+ 183us) server_negotiation.cc:407] Connection header received
0808 07:01:28.705564 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 07:01:28.705565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 07:01:28.705575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 07:01:28.705624 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 07:01:28.706333 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.707050 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.707999 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 07:01:28.708177 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 07:01:28.708222 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 07:01:28.708367 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 07:01:28.708398 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 07:01:28.708444 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 07:01:28.708589 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 07:01:28.708609 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 07:01:28.709888 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23830
I20260808 07:01:28.722859 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 23936
I20260808 07:01:28.732733 21617 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDMxMqG/build/release/bin/kudu with pid 24049
2026-08-08T07:01:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2912 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5771 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269962 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST